Security Unfiltered
Security Unfiltered
Patching the Unpatchable
Discover how technology is reshaping our lives and livelihoods.
Listen on: Apple Podcasts Spotify
Root.ioDisclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.
Follow the Podcast on Social Media!
Tesla Referral Code: https://ts.la/joseph675128
YouTube: https://www.youtube.com/@securityunfilteredpodcast
Instagram: https://www.instagram.com/secunfpodcast/
Twitter: https://twitter.com/SecUnfPodcast
How's it going, john? It's great to finally get you on the podcast. I know we've been trying to get this thing going for a while now. It's definitely been in the making for a while, and I very inconveniently had my second kid in the middle of it, so that hasn't helped anything, that's for sure.
Speaker 2:Thanks a lot, joe. I appreciate that and I'm really excited to be here. Timing these kid things out always kind of it's magic. You know, you never really know when they're going to come, but it's important when they do, so I'm glad we could push it around. I'm certainly not as important as that moment for you, so congratulations.
Speaker 1:Yeah, thanks, it's. You know, it's always like, at least with my first, my first born, and with my second one, now you know, they tell you one date and then it's like three weeks earlier, no matter what. So this time around with my firstborn, I wasn't ready at all. I didn't have the crib set up, I didn't have anything. And so with this one I made it a point like okay, this time around, like two months ahead, I'm painting the room, I'm getting the crypto and everything else you know. So I refuse to be unprepared for this one.
Speaker 2:Anyway, early is super early for the first one for that exact reason. You know you got to. You think you got a lot of time. You don't really know how the pressure is going to be. But yeah, I have three kids and they came at all different times, but the first one was actually late for me, which gave me a little more leeway. I actually did a good job of getting the room ready in advance. And the second one doesn't sneak up on you as much. You've been getting any sleep. You look awful awake for someone who's got new kids running around the house.
Speaker 1:Yeah, yeah, so I've been sleeping pretty good. It's the wife that does the night shift, and so she's going from nap to nap and you know, like when I, when I, when I do a lunch with someone or something, it's always like all right, let me bring you food, you know, because I know you're, you're the one taking the sacrifice.
Speaker 2:She's doing an amazing service. That's, that's hard work. So you got a good, you got a good partner there.
Speaker 1:Luckily she's a teacher too, right. So you know she takes her maternity leave, you know, right when the kid's born, obviously, or a little bit before, and she gets it all the way through the rest of the school year and then she goes into summer break, you know. So it's like, man, you're getting like six months, you know six months off from work, not like you're not doing anything, but still it's pretty awesome she's definitely got a job, but but still, it's pretty awesome.
Speaker 2:She's definitely got a job, but on her hands raising the kids. But that's really lucky, right? That's a luxury. You get to spend some quality time with your newborn and I can't think of anything better to do.
Speaker 1:Really, if you put it in the grand scheme of things, yeah, yeah, you know, I started my career, obviously, like probably everyone you know, going into the office, like like everyone else, right, every single day, five days a week, going into the office. And now that I work from home and I've worked from home since, you know, 2020, like everyone else, basically I just couldn't imagine going back into the office and how much I would miss, you know, with with my kids and everything I mean. I'm there. I'm there when my kids wake up. I'm there when they come home from school, I'm there in between you know like so good it's so good.
Speaker 1:Yeah, what a great time to be on that.
Speaker 2:Oh, there's no number Like it's the most precious time ever and, uh, the fact that you get to do that and do your job and and do your podcast and all these things is like a luxury, awesome thing. I had my kids a little while ago and it was go to the office every day and I was lucky enough to have a wife who really managed doing a great job with them and keeping the family on track with all the things, and I wish it was different. I missed some good time with them and spent a lot of time with them, coached all the sports and things like that. So it paid off in the end. But yeah, it's, it's. It's cool to hear that you're enjoying this time in your life.
Speaker 1:Yeah, yeah, I always, I always really do my best to like enjoy the present and plan for the future, you know, so, like that, that's kind of how I approach. You know just about everything right.
Speaker 2:It's amazing, like that's kind of how I approach you know just about everything right. Amazing Zen, perfect tip for the audience. Yeah, that's how we all should live.
Speaker 1:Very cool.
Speaker 1:Yeah, yeah, awesome. Yeah, john, you know I started this podcast. Give you a short background, right? I started this podcast because I wanted to talk to experts and hear how they got into the field. To experts and hear how they got into the field, because I remember when I was trying to get into cybersecurity, I had the worst time trying to get into it and no one would give me guidance.
Speaker 1:You know, no one really took the time of day to to like, tell me, like, hey, these are the top five things that you need to really work on. You know, like, even when I would get rejected by employers, you know, the only feedback I'm getting is oh, you don't have Splunk experience. Well, thank you, because that costs 200k a year for me to run. You know, in my home network or my company network. Yeah, I'm not going to be able to get that, you know. So I always start everyone off with telling how they got started, because everyone's background is unique, everyone's journey is unique, and I think that it's valuable. It definitely would have been valuable for me back then to hear someone else's story that I may be related to. Maybe I said, oh, that sounds like me, that sounds like what I'm going through and just given that little you know, motivation to hear hey, if John can go through it and be successful, maybe I can go through it too.
Speaker 2:That's, yeah, it's. My journey is kind of interesting. I think you know, build cybersecurity companies that's that's like what I've been doing for the last 20 years. So we start, and that's a lot about imagining a better future for you know, very important cybersecurity use cases Like the one I'm currently working on is how do I help organizations remediate vulnerabilities faster in containerized workloads like faster, better, more efficient, less work and energy expense in the company. That's what I'm currently working on.
Speaker 2:Now. The journey to you know, from sort of I've been around a while. I've been in the industry, like 25 plus years, right. The journey for me is the interesting one in that I've always been sort of a builder. I was working in startup companies and my own company since like 24 years old, right. So like very early on I had this entrepreneurial bent and I always kind of took my own path to thing, and I think there's a parallel we can help your listeners take along with them, especially in this day and age. So it's a very simple. I'll super summarize this.
Speaker 2:One of the early companies I founded was an internet service provider. Literally. I had an ISP that I founded that had dial-up modems. That's how long I've been around, right. Dial-up modems, there was no cable internet access. There was no cable internet access. There was no big verizon network doing this stuff. It was mom and pops with a t1 doing like internet access for the neighborhood and the people around you, right. And so we had to deal with like early stage network security stuff that like there was just not a lot of information about doing. So we ginned up firewalls on servers and did all kinds of things, and then checkpoint came out with their firewalls on servers and did all kinds of things, and then Checkpoint came out with their firewalls. So I kind of got pulled into being, like I'll call it, a specialty. Like myself and my founders were networking guys I was building, also building other networking products at a different company that I had founded. But basically I had to do security well to do my day job well, which was build a product that was network service provision. So I had to do security well to do my day job well, which was build a product that was network service provision. So I had to learn a lot about network security and to learn a lot about securing the infrastructure that we were building. So pretty straightforward stuff.
Speaker 2:Now, fast forward. That was a networking company and slowly but surely my career kind of transformed into being from I'll call it hardware networking and hardware building, hardware systems to building software systems. And along that journey I got. We built a software company. At that time we got a malware attack that kind of did damage and we were building software for other purposes, with software towards networking.
Speaker 2:But it was like the first time I'd encountered a meaningful attack on me that the root of it was software security. I couldn't get through that malware attack and so I really took it on as like a pet interest. I said you know what? That's never happened to me again. This is my new passion, and I literally at that moment flipped the corner and said I'm only going to work on security problems right now and I need to stop this. This is terrible. It revealed itself to me as a wow, this is a really dangerous thing for anyone, especially organizations that can be subject to this. You know the Internet was a thing, now right, and it was easy to mess up, and I thought, wow, I have some purpose in this, I really feel like I can make a difference.
Speaker 2:And since then I really not started thinking about anything other than I not stopped thinking about software security, web security, web application security, saas security, you name it and it sort of led me down a really fun, interesting and great career path where I've built now several big businesses and companies, both starting them from small and growing them to raising venture capital and building companies like I'm doing now, to running a half a billion dollar cybersecurity business in Cisco. And I would say the takeaway for that, like let's bring this down to ground right, like so that sounds like a lot of stuff happening fast. It wasn't. I really invested in learning. I studied everything I could.
Speaker 2:By that time, the internet had volumes of information. I read a lot of books like how to hack. I wanted to think like a hacker, I wanted to act like a hacker. So I took I'm an engineer right, I'm a curious engineer with good programming skills, with good hardware engineering skills. I know how networks work. I know how software works. I have background in engineering degrees and stuff like that. So I really made it my pet passion and then turned it into my business passion and business focus. And it started with the pet passion which was hey, can I learn how malware works? Can I investigate that and try it myself? And I'm telling you, if I had AI on my desktop to work with back then, it could have accelerated my learning a ton.
Speaker 2:So I would say, like, for anyone who wants to get into it, be curious, find something that you're passionate about, put in the work and effort to learn it, learn it really well and make it so that you can demonstrate that learning. And I think in today's world that's you know, like, make a GitHub repo of your examples. Like build something that actually demonstrates your knowledge or demonstrates the problem, or, you know and again, you're a guy who does blogs to help people learn like put it out there, put it on your LinkedIn, put it, you know. Like learn in the open and really get people to recognize that your passion's there and that you're putting in the energy. And I'll tell you, in cybersecurity you got someone who proves they know something, shows they want to work hard, can demonstrate their value. There's like a million jobs in cybersecurity that are unfilled around the world. I mean like I think you can actually build it yourself by rolling up your sleeves. And that's pretty much my journey. It's taken me a long way. It's really, really fun.
Speaker 1:Yeah, yeah, that's. It's a fascinating journey Because I feel like everyone knows what to do right or how to do it to be successful, but not a lot of people go down that road, right, right, right, you know you said you kind of mentioned that you're like a lifelong learner. You know, you really dove into this thing, you know head first and I really relate to that for sure. Like security isn't just something that I do nine to five. Right, in case people didn't realize, you know, like the podcast hints it a little bit. But you know I'm also challenging myself significantly in other ways. Right, so I'm getting my PhD and securing satellites with zero trust to prepare them for quantum encryption? Right, like that. All three, all two areas of that category, right, zero trust, I know, yeah yeah, yeah, that's the only thing I'm bringing to the table.
Speaker 1:I know zero trust, but satellites never touched them, never looked at them, never researched them. Quantum I didn't even want to look at it. I didn't looked at them, never researched them. Quantum I didn't even want to look at it. I didn't. I mean, even to this day. Like you know, I finished, I think, like my 50 page 50-ish page section of my dissertation on quantum. You know, just giving a base foundation, doing my best to not sound dumb while talking about quantum. Right, I just finished it, so I'm not even trying to think about it now.
Speaker 2:For the rest of the year, but doing those things— Congratulations, by the way, that's a huge part of the lift on getting to that point in your education, right?
Speaker 1:The dissertation and—. Yeah, my research plan actually just got approved, I think like maybe last week, and I was torn with emotion because I was like man, I kind of hope that they would have like denied me and said, no, this will never work, you know, just leave the program. But unfortunately they, they approved it and so now, you know, I told my chair, I was like it looks like I actually have to do this stuff now.
Speaker 2:Congratulations. I hope that I'll be calling you Dr South here next time we do a podcast together. That'd be super fun.
Speaker 1:I'm sure we'll do one more before I actually get that accolade. Okay, cool, quickly there. And I think of a few friends of mine that didn't put that time in and now they're unemployed, and now they're in a situation where they're looking like, you know, the other million, two million candidates out there for these jobs, when companies really want someone that is passionate about it, you know, and it's doing the little things that show that interest, that show that knowledge and that passion outside of your nine to five. Like you said, a GitHub project speaks volumes. It speaks so many more volumes than an interview ever will.
Speaker 2:And it's also an open book on your work that people can look at and collaborate with you on, and even see into your soul, so to speak, on how you think and what you do, and that's like the best living example of your work, right? I'll mention one other thing. I was lucky to get mentors early in my career, like people that invested, you know like believed in me and invested time in me to help me, whether it was a person at my company that I had been working for, like my partners in building companies, or we got acquired several times. I've had five of my companies acquired, so, like one of the early ones, I was still a young guy. I was like you're not even 30 yet and we got acquired.
Speaker 2:And I saw it happening in this organization a lot, where younger people with ambition would easily go under the attention of someone with experience and they would take them under their wing. And I had the luck of having that several times through my career, through these acquisitions, or even through customers and partners, me. I've had this happen lots of times, several times, where somebody will LinkedIn me a person that's just out of college and ask me questions about my career, what I'm doing, or say hey, john, I'd love to talk to you for 15 minutes. See, like I want to go down this path. Can you talk? And I do it like I do it and not as much as I'd love to do it. But I, when it comes up, I'm like you know that's a chance to give back. And when it comes up, I'm like you know that's a chance to give back. And I think if you ask people to help you out, they do, and so I think that's another lesson for people like who might be trying to go down the security career.
Speaker 2:It's like you know, go to a security meetup in your area and just talk to people, ask them what they do, be curious, put yourself out there and say, hey, I want to get into security. Can you give me some advice? And I bet that pays off hugely because join the Cloud Native Compute forum, join OWASP. It's like 50 bucks, right. Join OWASP. Be curious, show up to the meetup in your townago's got a great like oauth community right and go be around security people. They'll guide you. Uh, contribute to an oauth project. It's out there to do like. I know, for instance, steve springett like a close colleague. He runs open oauth dependency track, it like 1,800 different organizations that use it like a mainstream product. He's in Chicagoland, that's where he lives. Go to the OWASP meetup, talk to Steve. Go, show that you know something about Dependency Track and I bet you one of those 1,800 companies would love to know your name.
Speaker 2:Simple stuff like that.
Speaker 1:Yeah, yeah, that is actually really true. You know, putting yourself out there is probably 90% of the battle, right, Like I remember when I was getting started, I would go to those meetings just to meet people, learn something, you know, and I would bring a stack of resumes with me and I'd hand them out. You know, like what do I have to lose? Someone going to turn me down? Exactly right, I went to one of Someone going to turn me down.
Speaker 2:Exactly right. I went to one of those in New York City probably six months ago and half the audience were college kids from one of the universities in New York. They were all graduating in the cybersecurity program from that university and they were out there talking to like grizzled old professionals like me and all the guys there. But you know they've linked in with me now. They send me notes. I have like a. You know, you meet someone and you get a little like investment in, in, in their, in their name and what they do, and you're like, ah, I saw that guy at the thing. Yeah, you remember it. That little bit of ingenuity it goes a long way. It makes the people take notice.
Speaker 1:Yeah, yeah, yeah. It's interesting also leveraging, you know, your network right, like building that relationship, really really diving into that. I take a lot of pride when I make a recommendation, you know, to someone for another person that they've never met. You know, and I give that recommendation and you know the person I'm giving it to says, you know, automatically it's a yes because Joe recommended them Right, like that's that really. I feel like that speaks volumes, you know, to my own reputation within the industry where it's like, you know, when I say something I really do mean it, I'm not just saying it just to say it Right, and I personally take a lot of pride in that. You know I like helping out people and I always try to go out of my way and do it.
Speaker 2:I think that's a paid forward moment right. Like you, especially if you're invested in the industry and the cause, I don't take cybersecurity as just like a day job. I really want to make a meaningful impact on the future of cyber safety for my money and really is a battle of good and evil. I hate to be too trite with that, but it is really that right and so paying it forward so that you help the next generation of people coming in the industry to drive the good guys forward is good work, so something to be proud of. I think helping people out in this field is, is, is.
Speaker 1:Yeah, absolutely. Well, you know, John, talk to me about the problem that you know you identified. That led you on the path to creating root IO and the solution and the product behind it.
Speaker 2:Yeah, it was a very practical and very practical and close to my heart problem. So, as I've mentioned, aside from starting startups with three guys, four guys and an idea, which we did with Root, I've also run some pretty big businesses inside of Cisco and inside of CloudLock and others no-transcript infrastructure and cloud-native applications running in AWS and in custom infrastructure, etc. We've been building containerized software for a long time, since the dawn of containers and Kubernetes. We've been building on that and building that really like, for instance, applications that were in my in the product area that I ran at Cisco had, you know, like millions of users and hundreds of thousands of companies using it. Cisco Umbrella was one of the products and it was a really popular product.
Speaker 2:Security is paramount there. Right, I take seriously the products I build for the value they create, but I also take seriously the fact that that's a security product. It needs to be really frigging secure, right Like super secure. So it's always been near and dear to my heart to do a great job with AppSec for these applications we build, and one of the challenges I had at Cisco and at other companies we've been building cloud-native workloads is how hard it is for organizations to get a handle on the software supply chain, security and the vulnerability management for these applications. So specifically, what Root does is it helps organizations remediate vulnerabilities and lower the attack surface on containerized workloads. So you know your container image is the embodiment of your software supply chain, like it is the running implementation of the software you build and the software you use. So if you look at any container image, it's like 95% open source software Linux, python, python libraries, etc. Like the world of open source provides and it provides all that free software free as in speech, not beer, it's not free. You know you still have to pay to use it effectively, like it costs you to secure it but it didn't cost you a license fee and ensure you didn't freely use. I always say that software is more like free puppies you get them and then you got to do a lot of work to make the puppies successful, so to speak.
Speaker 2:But anyway, that challenge of putting I'll call it vulnerability-free and low-risk profile containers into production was really hard for these companies to do and it puzzled me. Like in 1,000 engineers we had, like you know, a massive organization and we were going through things like FedRAMP compliance and we were doing a lot of like really strict security management on these things. And I'm like why, with all the resources we have at Cisco and all the know-how and the great engineers we have, why is this so hard to do? We get our vulnerability scanners scanning these containers, we get these long lists of vulnerabilities and then we go okay, let's try to get our handle on how to fix these. None of the tools could remediate. So, first of all, the biggest problem, like problem number one a lot of scanners tell you about vulnerabilities and none of them do anything to fix it.
Speaker 2:Basically and I think that's largely still the case today and then changing out the operating system or moving to a new version of the operating system. Operating systems are where most of the vulnerabilities come from. By the way, it's probably some in the know-know, but it's like the base image typically has like 80% of the vulnerabilities in it. It's getting to be the point where libraries also have lots and lots of vulnerabilities. But you can get a lot of banks for the buck if you just fix up the Debian layer in the container, or the Debian layer plus the App Foundation layer.
Speaker 2:I saw our engineers struggle mightily to get any progress in that, like why can't you change the OS. Well, if we do that, this dependency will break or it's going to take us. You know we can't build the next three features that you want to deliver in six months to get our customers happy if I do that job. So which one do you want me to do? Do you want me to fix the, make the software secure, or do you want me to ship the features? There's always this. And then, of course, you got SEC engineers like you know, people that I, like you know, like I revere, like saying you got to fix this stuff, man, I'm not going to be able to do the Fed ramp. All these pressures from compliance and security governance, which is good, means that you're building secure stuff was at odds with my roadmap. And then, when you get the engineers to work on this stuff, they don't like the work. They'd rather make features, not fix debt.
Speaker 1:And this is security debt, right.
Speaker 2:So it's like they don't want to work on fixing these things, and so I searched around for a lot of solutions. There was nothing and we started this thing like four or five years ago and what I realized was, look, if I'm going to make I kind of went on like first principles approach to this Me and my co-founders were like what would be the most beautiful experience that you could ever have. That would make, that would be cost effective. It would solve the solution fast. Meaning like organizations wouldn't have to have some heavy lift, they could work in the same workflows that they have now. Meaning like they didn't have to change their build tools or they didn't have to switch operating systems, like from debian to alpine, or they didn't have to like do a heavy lift and I didn't put a bunch of shift left work on the place of developers and I know they don't like to do it right, they don't want to fix that shit, they want to build new shit, right. So I'm like what is the perfect solution? What would really move the needle and make it easy for organizations like mine to have this? Like low effort, high efficacy, security for container images, get to zero vulnerabilities, but not create a new pile of work to get that job done, and so Root AVR was born.
Speaker 2:This product we have, and at the core of it, it gets to the heart of the matter very quickly. What we do is we patch container images. We patch them so they get to zero vulnerabilities, and we do it fast. Fast forward to now. We came up with that like nirvana solution that I wish I had when I was working at Cisco. Took a while to figure out, worked really hard, got a bunch of smart people together, me and my co-founders. I've been passionate for this thing, but in the end we achieved that. What I said, which is can I build a solution that can take any container image that we have and convert it into, patch it into a zero vulnerability image in five minutes, in less than five minutes? And that's what our product does it makes container images go to zero vulnerability.
Speaker 1:So, okay, that's really fascinating to me, right? Because I'm someone that you know has been on the security side, that was over there buying developers and engineers drinks and lunch just to get them to.
Speaker 1:I know it very well, job that I really need them to do desperately. Right? It's like, hey, I can, yes, I can technically do this, like I know how to do it and I probably even have the access. But please can you just do this for me, like put that feature on hold for one sprint. So talk to me about how that's accomplished, right? So from what I understand from container images is essentially, you know a file, right, and then you're probably and correct me if I'm wrong you're probably updating the different components of that file, and then when the container because containers are supposed to be short lived when it, you know, goes down and then it starts back up, it's using that new patched image and it kind of just rolls through the environment, like that.
Speaker 2:That's generally right. I'll dig in a little bit more. I'll go like a layer below that. That's like directionally very right on. Again, I mentioned that at the heart of our solution is this ability to patch images and we do that on an image and it's your image. So at its edges, our systems, kind of schematically, it works at a high level like this you attach us it's a SaaS service, so it's a patching service you attach us to your registry where you keep your images and typically we start by operating on an organization's base images.
Speaker 2:Think of, like Node on Debian or Node on Alpine or PHP on Debian or Elasticsearch on Debian. Like you have these kind of app foundation images where you have like a Node or Python environment or you have, like I'll call it an infrastructure applications like Redis, postgres these are the kinds of images I'm talking about that you start with and that cleans up, by the way, when you get these to be clean, that takes a ton of vulnerabilities out of the environments, because these images get propagated over and over again. Right, be clean, that takes a ton of vulnerabilities out of the environments Because these images get propagated over and over again. Right, you build your as a company. You build your images on these foundational images and you derive a lot of your microservices from the same base images if you're doing your DevOps homework right. So we focus on those. I'll call them foundational image sets. They're the 10 or 15 or 20 core images you use to build your SaaS from and you derive from. As you know, with images you can in Dockerfiles and other means you can derive from so you can do like a multi-stage build.
Speaker 2:The first thing is you do, you build your base and then you add more layers to it as it goes, and that's kind of a pragmatic way to organize around these base images. So we plug into your registry, so you just put a registry integration into our system. You can tie us into your CI CD pipelines. You can trigger us to run whenever you build a new, whenever you want us to create a new base image for you or I'll call it AVR, your base image, patch, your base image. These base images often come from places like Docker Hub or it's something somebody generated that first Docker file for. But a lot of times they come from community sources or public sources and then they get modified. But we start with those images and then, if you want us to act on it. You just tell us to go and act on the image by triggering us with your CICD pipeline or through our UI, and I'm happy to do a demo later on in the session to show you exactly what it looks like. But once we run, we literally pull the image into our system, into our SAS. We open it up, we analyze its components.
Speaker 2:I'll say largely we use a simplistic model, but containers are layered. This is because they have something called a layered file system, right, and the base image lives in whatever layer was specified, and so on. You build up some containers I've seen that have as many as like 100 layers, but on average they have like four or five, six, eight, something like that Small number of single digits number of layers. So in a simplistic model, most containers that go to production have like a four macro sets of stack. It's like the operating system or the very base layer. It would be like Debian Bookworm, right. And then you have another layer. I'll call it macro layer. It could be multiple container layers.
Speaker 2:But the next thing is like application foundations or applications. So that's where Redis would fit in to the container stack. Or Python, the app foundation, the library, python, right, like Python you might install on your computer with its basic packages. Layer three of that stack would be like the Python libraries I need for my requirementstxt to actually build the application that I'm trying to build. And the fourth is called first-party code. It's code you write that uses all that stuff just like you do it on a computer but in a container. We worry about first-party code. It's code you write that uses all that stuff just like you do it on a computer but in a container. We worry about third-party code, those first three layers of that stack, making that secure and patching that.
Speaker 2:And, as you mentioned, we apply patches. So we apply two different kinds of patches. Our system can automatically apply patches to those first three layers by simply doing upgrades, and we have a patch planner that knows how to do that. In a way that's very low likelihood of breaking your application. We look at the composition of the container and we choose upgrades that are safe. That's kind of a cool tech that we've created that kind of.
Speaker 2:Does some estimation on that? And then, even if you do a perfect job at that, there's going to be lots of vulnerabilities left over, because many vulnerabilities don't have patches in the ecosystem. Debian is notorious for this. About 70% of all the containers in the world are based on Debian that run in cloud-native environments. There's a large number of vulnerabilities that are critical and high that Debian does not have patches for, because they've decided to defer patching those to future versions of their operating system and they kind of leave it up to the community to figure out. But no one does so.
Speaker 2:The other part that we do is we go make patches where they don't exist in the world and we have a very big library of patches we've developed you can't find anywhere else. Really, wow, library of patches we've developed you can't find anywhere else really. Well, we, uh we've built an agentic ai system that that augments our research team and it does this extremely efficiently and extremely accurately. We've invested a lot in automation and a lot in agent technology over the last year and a half to make use of the cutting edge of ai tech to do this. It's a really complicated thing, meaning like making these patches requires, well, we patch times, hundreds of libraries and it would be like nearly impossible for engineers to understand all of those. Like we literally have to understand all the source code in those libraries, like it's hundreds of open source libraries.
Speaker 2:To patch, expert knowledge required. We have a great research team of expert patchers, guys who know how to do this work really well and they like doing it. The first thing is nobody likes to do this work. I got guys who like to do this work and we've made agents that love to do this work and it turns out that Agentic AI is really good at this problem. For instance, claude Anthropix agent. They've spent a lot of investment in making Claude a really good coder and it turns out this is one of those coding problems that we figured out how to make these AI tools be really, really good at.
Speaker 2:Now, obviously, we know how to do it and we're really good at it. We have lots of examples and we have lots of training data and we have lots of things. But at the core of our system is this ability to create patches where none exist and then patching a way that doesn't break things, be really surgical with all the upgrades and patches, and then, third, our system automatically applies them to the container image by creating a new container layer where all the patches go. We then generate, we build. Once we figure all that out, we get all those patches there. We then build a new image and then push it back to your registry and we just tag it with underscore, root, io and the combination on an average like, say I can show you later a Python image that we do this for the amount of hours, it's dozens and dozens.
Speaker 2:If you could even figure it out like we can patch an image in three minutes, that would take any team like literally any team with any other competitive solution or approach days, maybe weeks, and our system can do it in like I'll show you while we're on this call. Now that's the culmination of a lot of really cool research and work and engineering and building this massive software factory in the back end of our system. We've got a SaaS platform that makes it really easy to interact with and connect it to your world. Everything's got APIs, but that's pretty much the high level thing Container in, analyze, plan, apply patches to wherever we need them, reconstruct the container, emit the container and you're done. It happens in about three minutes For the average container. Your mileage may vary depending upon the size of the container. Now, the other thing I should mention is that as soon as we see your image and that's your image, by the way, I didn't make you change over to my image. That's your image.
Speaker 2:Whatever image you chose to direct us to try this on Competitive solutions, image whatever image you chose to direct us to try this on Competitive solutions, most of them are image libraries. They sell you a bunch of images. They're super expensive. The way they deliver them is super expensive for them to do. That's why they pass those costs onto you and it requires you to rebase your images. So to start with those solutions, you got to go back to your engineering team and go hey, listen, I got a great solution. Everything's going to be great. All you need to do is re-engineer all the images. We have to use this new operating system. It's going to break everything in the beginning, but that's okay because as soon as we get over that massive toothache, things are going to be good. But they actually don't turn out good because many applications can't port. Like, if you're on Debian or Ubuntu, you try to go to Alpine, stuff just breaks and you've got this massive lift. It takes you months to get started with those solutions. This was also one of my goals.
Speaker 2:I want anybody to be able to start in five minutes and with our solution there's none of that. You start with your images, the stack and applications that you love and already know. Your engineers are all tooled up on it. You have all your DevOps scripts and languages and all the things lined up. Your cicd pipeline understand everything that's going on. Your scanner understand everything. We start with that image. We patch it to zero. You have your image but better in five minutes. And so you know, I checked, oh, we checked all the boxes on our like wish list of how we could make this good for john amaral. You know, four years ago at cisco, cisco, and here we are doing that Now. As soon as we see your image, if any new vulnerabilities come up because we know your image, we go patch it. We find, like as soon as a new one comes, our software factory and our AI agents kick off a process where they go figure out what to do automatically. So you've been in security a long time.
Speaker 2:Patching SLAsas, right, are pretty strict like depends on on the regime, but you know, with federated it's like seven days for a critical um. You know what do you do when no patch exists. Well, you're kind of like you got to do a polam. You're going to be on the hook to like for con, bond, right, right, etc. Our, our, our autonomous software factory has always got your back. So it it's like you're you know. It's like you take the engineers out to lunch. You try to get them to like pay attention to the security stuff you want done. Our system doesn't need lunch, it just needs like electricity and it wants to solve all your patches for you. You don't even have to take it out to lunch. So that's how it worked. You know questions, comments, thoughts. I'd love to hear your take on all that.
Speaker 1:I mean, I think it's really, really fascinating. You know you're kind of teasing me here. Now I want to dive into the tool.
Speaker 2:Let's go, let's go. I'm going to do a screen share. Good thing we tested this earlier. I know it's going to work, so that's cool. I'm just going to press, go on the screen, share and, uh, yeah, when you said a screen share earlier, I was like, ah, we need to test that, cause I've never done that before. All right, we're forging new ground on the uh on on this podcast. I'm going to do uh, this guy right here. Nope, it's not on that one there we go Share.
Speaker 2:Hopefully it works twice. Are you seeing my screen? Yeah, cool, and hopefully it's zoomed in enough so the audience can take a look. All right, so I'll get in and certainly stop me, joan, I don't have. This is covering your face now, so I wish it wasn't that way, but it is Just yell at me to stop so you can ask questions as we go. So basically, there's a bunch of parts to this solution. One is we've got this dashboard that's trying to, you know, give you a high-level view of all the good work that this solution does for you. You can see and this is just a test account I test all kinds of containers in here, but in the profile of containers I've done, you see that I've had a 73% vulnerability reduction across all issues. It saved like 77 days worth of work, 270 vulnerabilities that are remediated. I sometimes throw stuff at this. That are really hard tests for it, but in general it works very well at remediating and you have an easy way to see what's going on.
Speaker 2:The system really, you know, kind of runs from this point. This is the remediation tab. This is where you get started. It allows you to connect a bunch of registries, so we have pretty much every registry under the sun. As I said earlier, first thing you do is you tie us to your registry. If you want us to work on your private images, we also work on public images. You can start by picking something on Docker Hub. That's what I'll do for this. But when you connect this up, you just put a registry token here, a security token, and then, once we have access, we write access to that registry. When we remediate an image, we just push an image right back to that same registry but tagged with root IO under it For public images.
Speaker 2:This is like a super simple way to test and anyone can sign up and use this platform. There's no gating on it. You can go check it out yourself. You can do all the things I'm showing you here without any intervention from our company. We don't ask you to call us for a demo. You just sign up, try it and it's easy. I'm going to pick Python. I talked about that earlier. You just select one of these images or you can pick from whatever tag you want up on Docker Hub.
Speaker 2:Today we support, as I mentioned, debian, ubuntu and Alpine images on AMD64. Pretty wide support setup and then you just click the remediate button. It's going to bring up a modal dialogue here that kind of tells you what's happening. So we first pull the image, we scan it. We have the trivia scanner under the hood. We also create an SBOM of that image. Once we do that, we go into this evaluation mode.
Speaker 2:We look at the container composition, the packages, the vulnerabilities and we make that patching plan. I talked about where we do patch selection. From the upgrade list try to select patches that have a low semantic version difference from the package that's in your image. But our goal here is to kind of like preserve the packages you have from a major and minor version perspective and bring I'll call it very small change upgrades to it, just targeting those vulnerabilities. Now, the goal of this is not to upgrade functionality, it's literally to patch away the vulnerability. So we're very targeted in that and of course we also select from those root patches, those root patches that we've generated that we can use with the same mindset. It's like bringing these patches to bear to literally target elimination of vulnerabilities so that your risk profile in the container and your ability to meet compliance for vulnerability counts in SLAs is made easy.
Speaker 1:Yeah, it's fascinating because I'm wondering how long it took you to train the AI model, to build up to the point where it kind of knew what to do right, where it was becoming more effective for your use cases. Can you talk a little bit about that, potentially?
Speaker 2:We're pushing to roots registry. That's what it should do for a public image. So now it's scanning. So let me take off from here and then I'll answer your question in a minute. Okay, so it's scanning the image. Now it pushed it to an internal registry that we have, an ECR registry, where we create a temporary location for it. This is a public image, so it didn't have your registry to push back to. We scan it again after we do all the patching and then we then present the findings and a link for you to be able to go and retrieve the image. So part of this is we generate VEX and S-bonds for everything we do. So the VEX statements actually have a very detailed record of everything we patched, all the new packages we applied, and you can use that VEX file to inform your scanners of these package changes, because natively, most scanners don't understand these package changes as we've generated them. Natively, most scanners don't understand these package changes as we've generated them. We make it really easy and transparent for users of our system to be able to do that.
Speaker 2:And voila, here you go. We've got an image that had one high and critical and I'll show you more about this in a second down to zero highs and criticals. Now that image was a curated image that already had like a bit of a risk surface reduced. And then here is a full command for Docker that you can go and grab that image yourself, drill in and get the full report and further. It just didn't get rid of the high and critical, it got rid of all the mediums. So this container has zero highest criticals and mediums. This is a very typical result for the image, for these operating system images that I've shared. It could be Python, node, whatever, php, but we have very good success across a whole bunch of images. I'll show you the kinds of images we operate on and we have a big list of those that we test on and offer, as I'll call it, convenience images for people. But we don't sell images. We sell this service. You can run this on your images. If you have a Python version that's on Debian that you use for your base or node or you name it, you don't have to rebase. You just run our tool and you get a new image that has much less vulnerabilities Behind the scenes.
Speaker 2:Here we added 18 of these root patches, these custom patches that we developed using our agentic AI system and our really fabulous research team. And something to note I mentioned that these semantic versions are very close and in fact, this libcap2 vulnerability has no patch in Debian. That's why it's a root patch and we actually provided a patch to the exact library you had with our root extension and we probably only changed a few files in kind of a small number of lines of code probably less than 100 lines of code is typical that we'd have to change. It's very surgical. It doesn't change anything with the package that would make it any compatibility issue. All of the capabilities, file signatures. We regression test these libraries so we know they work exactly like the version did prior to patching. So it's an exact fit, except the vulnerability has been patched away. And that's what we do.
Speaker 2:We work on the who's who of images. Just one more thing, sorry, you can come in and look at our image catalog. These are all images that we have done extensive testing with to show that our AVR works well across those operating systems. In fact, these images are also. They're not patched by us, but we do offer these as images that we've curated a bit because, um, I think if you just go download these from, like you know, public registries they'll have. Like you know, some of them have hundreds of vulnerabilities. We've done a service to the, to the world, to make these, like you know, something much smaller and much more, um, I think, lower vulnerabilities, because I don't think you should start with that 500 bones but then you can run these and patch them down to near zero or zero with our system. So if people want to use these, they can, but it's not necessary. You can just use what you got and run it through our system, and that's how people do it.
Speaker 1:So with the patches that you showed on the previous screen? So with the patches that you showed on the previous screen, are you essentially? Is that considered backporting a patch into?
Speaker 2:the package Exactly right. That's a backported patch and our agentic AI system and our researchers. What they do is, for instance, for that one I showed you, lib, libcap, and this is true for every one of those patches. Our research, we have two, two agents. We have a research agent and a patching agent. The research agent goes out and researches the vulnerability and it researches the effectively. It scours the internet. It knows you know good starting points to, but it goes and looks for code commits to this ecosystem of libcap. So it could look at the maintainer's GitHub repo where all source of code comes from. Or it could go look in Debian's repos you name it. It could look in Ubuntu's repos Theans repos you name it. It could look into Ubuntu's repos.
Speaker 2:We look for places where we find a FIX in a later version of that library or we look for a FIX that may exist that's never been merged with that library, and these happen in two forms. And then backporting happens when we find those and then we basically analyze our patching agent. Once the analysis is done by the research agent, the patching agent takes that research and it does code analysis on a place where that fix exists, maybe a newer version of the software say LibCap was version 1.3.5. If 1.2 has it fixed, but 1.3.5. If 1.2 has it fixed but 1.35 doesn't.
Speaker 2:We look at 1.2's code. We have our patching agent examine it, understand the changes that were made related to the commits around that fix. We deduce all of that. We then look at the code in 1.3.5, and we fit those changes into that code base, build a set of commits for that, merge it in and then run extensive tests on that library to make sure that we didn't break anything and also verify that the patch worked, meaning that it actually did mitigate the vulnerability. So there's a lot of testing and there's a lot of this backporting and code identification.
Speaker 1:But that's, exactly how it works.
Speaker 1:Is there? I mean, maybe I'm just not as familiar with like container security patching overall, right, Is there a possibility of running into or inducing issues with this kind of style of patching and maintaining those images? I ask kind of twofold, right, if something does go wrong, how do you go backwards? And two, is it even like reasonable to think that things would go wrong, which sounds very weird from a security perspective? Right, because in security you're always assuming something's going to go wrong, something's going to break. You know there's going to be some sort of issue, right? So you always have to have like that backup plan in your back pocket just in case? Yeah, so that's a good question back pocket, just in case.
Speaker 2:Yeah, so that's a good question, a good multi-part question. I'll unpack it. So first of all, as I mentioned, we're extremely surgical with these changes. This kind of patching is not I'll call it exotic or foreign. This is what you do when you need to patch stuff. We're simply using upgrade mechanisms that are built into package managers, etc. The sophistication here is figuring out which patches to apply and also building patches. It's not uncommon I think maybe that's an understatement.
Speaker 2:It's common and universal that folks pick up new container images when they want to have upgraded packages. I mean, if you got your image off docker, how do you wait for the next version? You got your patches there. You take them right, or you run apk upgrade or you know, or you know you run the upgrade on the, on the docker file, you build a new image and that's your new image. So it's not uncommon to upgrade these things. It's is all third-party code, so it's very unlikely it's mostly never the case that people have these libraries as something they build. When you use Debian, you don't start with Debian's source code. You start with Debian's images, so you start with Debian's distribution. So patching this way in containers is not foreign at all. It's actually quite natural, it's normal.
Speaker 2:As for breaking, that's the. We don't break images, basically because we really manage to change scope, and that's the key. That's the art of this. Backported patches have really high reliability and these semantic version differences like these are just the only code that changes is just the code that's needed to fix that patch. And when we select upgrades, we also look for really small semantic version differences and that means that these are just security updates, which are, you know, technically with a good provider. That semantic version difference means that they've only introduced security fixes and the implicit guarantee is that those don't have any functional breaking changes. Now we do a lot of verification on this stuff to help make sure that that's the case, but in general, that's a good assumption. So we're doing kind of normal things, we're doing them very safely and we're really good at doing it well.
Speaker 2:Now how do you go back if something breaks? Well, you have the original image. We only patched the one you gave us right. So you've got our image. You've got your old image. If you've got DevOps maturity and you run tests and staging and all that, you put our image into your normal DevOps flow.
Speaker 2:As I said, people turn us on through CICD where they run us and they and I'll stop sharing Sorry, we're sitting here, so you just run us through your normal testing paces and and away you go. So again, I mentioned that we put all these fixes in one layer in your container, so literally like layers one through n are your layers untouched by us. Layer n plus one is our stuff, so everything is isolated there. The cool thing about the layered file system in images and the way they build is you can kind of modify any of the layers from any layer, so to speak. That's an oversimplification, but all our changes exist in that one layer. If we break, you can try the old image. It's easy to figure out what's going on and then we have some pretty cool ways to debug if you run into problems. But generally our customers don't have any problems. They're good.
Speaker 1:Yeah, that's fascinating. You know, earlier on in my career I dealt with a lot of backporting patches right and working with my devs to get them backported in. And then you know the scans are still failing right, because the scans are kind of dumb in what they're looking for. They're looking for that version change. Well, if you backport, the version doesn't necessarily change.
Speaker 2:That's where the VEX comes in. So we use these VEX files to transfer this and lots of scanners can take VEX as input now, so that basically informs them about the package changes and what we changed and that's also a transparency measure for you to be able to use that to, or users of our platform to be able to understand our changes. I will also add that every fix we do, we subject it to standard open source practices. So we publish all of our fixes, all the code we change, into a publicly available GitHub repo. It turns out it's really hard to do anything with these patches because the build stuff is really like we've got a massive software factory and to reproduce all these patches in a functional way from the source code isn't that easy. But we put it out there because we want people to see what we change. We want them to be able to review it if they want, and our customers want that. They want to know that we're transparent.
Speaker 2:We're not a black box, we're an automation system and we do a lot of heavy lifting for customers to get those backports done. I'm sure that your experience with those backports is it takes a long time to build them. Engineers don't like to do it and it's fraught with risk and I'll call it risk from getting engineers to deliver is hard because it takes expertise to know how to backboard things. We're really good at it and our agents are really, really, really, really good at it. I don't know what your memory is, but our agentic system can produce backboards in like 10 minutes. Wow, it probably would have taken engineers to do the set I did. I know how long it takes. We did a bunch of them manually because we like to learn. You know, the average backport takes three days, four days of an engineer's time, maybe more.
Speaker 1:It depends how complicated.
Speaker 2:It is For really good research engineers that's like. It can be hours to days, to weeks, depending upon complexity. Some of those patches we generated have a dozen, two dozen file changes, five, ten commits that we got to figure out how it applies to that older version, and our system is very excellent at that. Back to your question, though, I would say, yeah, we're getting integrated with scanners, so they naturally know about our packages because we put that rootio and we also have a feed that matches up the vulnerability fixes and the call it the, the effectivity statements, like.
Speaker 2:We fix this by doing the following into a feed and scanner vendors can take those in and inform their databases. So that's one way to tackle it and this is a very common way to integrate with scanner vendors so they know about this sort of thing. And then our VEX statements can be imported into scanners Trivia is excellent at using this Docker Scout's another one that's excellent using VEX so that you can inform your scanners about what we fixed. And, of course, our scanner in our platform, which is Trivia, uses the VEX statements as well.
Speaker 1:Yeah, it's fascinating. It's an interesting way of approaching this complex problem that so many people just want to run from it, right. They kind of want to dodge it, and more and more of the infrastructure on the internet is running off of containers, and so people in security are just like you know. They're kind of at a loss, right, because they don't have something that they can say. Hey, you know this isn't going to add an hour to your day or add a week to your sprint or whatever it might be. You know you can still deliver. You know your feature requests, your milestones and whatnot, as well as maintaining security in the product that's. You know. It's an interesting proposal, for sure.
Speaker 2:Yeah, and what you didn't see when I ran our tool and this is exactly what our customers experience is I didn't have to ask some engineer to change anything. All you need to do is plug us in. Engineer change anything, all you need to do is plug us in. We literally have customers that do this for their 10, 20 base images, plug us into the CI CD pipeline, get onto this SLA easy button Because, again, we're fixing all the new vulnerabilities that come too and you're getting those as the service runs.
Speaker 2:And that's really the core of the service. The core of the service is making it easy for you to very quickly get your images to a really much better place, like near zero vulnerabilities or zero, as I showed you, but then making it so that as vulnerabilities come, we're taking them on and fixing them as you go automatically. So SLA is kind of the core of the service. They get through doing this on all their images in a week, less than a week. Then most of it's just them integrating it into their workflows and getting CICD set up. But we have customers that literally plug us into CICD in four hours, run all of the images, get through all of the testings and be done in the same week. Yeah, no engineering resources required, and this will save them hundreds, if not thousands, of hours a year.
Speaker 1:Yeah, it's, that's definitely fascinating. That's really interesting to to dive into. Well, john, you know, unfortunately we're we're at the top of our time here and it's been a fantastic conversation. I really appreciate you coming on.
Speaker 2:I loved it. I think I really appreciate that you do this and kind of put great information out there and excited that you are kind of caring for the community of security people out there. I think this is a wonderful service. So great to meet you and do this today and happy to help any way I can.
Speaker 1:Yeah, absolutely. Well, I really appreciate that. Well, before I let you go, how about you tell my audience you know where they can find you if they want to reach out and connect and where they give their root?
Speaker 2:Sure. So you can find us at wwwrootio and you can use the application just like I did, with no hassle at approotio. Approotio, r-o-o-t. Like you know, the root the tree, and if folks want to reach out to me, you can just hit me up at john at rootio. I love connecting up on email. Shoot me a note if you're interested or you want to talk.
Speaker 1:Awesome. Well, thanks everyone. I really hope that you enjoyed this episode. More to come. Thanks guys.
