Dwayne 00:00:00.000 But the sad reality is that's human speed. Everything I just described describes a human speed situation. Breakout times, according to CrowdStrike now, are dwindling down to there was one attack they saw from the time of the attack actually broke through to the time exfiltration started was 27 seconds. How did it do that? How did it do the lateral movement that fast and unlock and start exfiltrating that fast? It harvested all the credentials it could. That's happening at machine speed. This is what drives all these stupid supply chain attacks, is how fast can we dump your creds? How fast can we publish forward, and how fast can we exfiltrate anything that's attached to those systems? And then self-propagate along the way This is DevOps Paradox, episode number 376, Secrets Management for AI Agents
Darin 00:01:43.424 Viktor, how often are you using secrets managers?
Viktor 00:01:46.294 All the time. Every day. Many times a day
Darin 00:01:49.664 Is it just you using secrets managers?
Viktor 00:01:53.264 You mean are my agents using secret managers? The answer is yes. The answer is yes How can they operate without knowing my secrets?
Darin 00:02:02.062 They could. We just don't want them to. On today's show, we have Dwayne McDaniel on from GitGuardian. Dwayne, how you doing?
Dwayne 00:02:09.734 I'm great. Glad to be here
Darin 00:02:11.864 Does that little interaction sound even closely familiar to what should be happening in the real world?
Dwayne 00:02:18.274 That's what it should sound like, that, "Oh yeah, I'm loving my secret management tools. They make my life easier." And the reality is sometimes they do. There is a happy path where I know teams that don't think about credentials because they've got that figured out. But the vast majority don't really have that thought. I've never heard anybody have that conversation out loud
Darin 00:02:42.494 Why do you think that is?
Dwayne 00:02:44.164 The short, simple answer is developers want to go fast, security tools want you to go slowly and correctly. And when you're in a hurry, developers are rarely getting paid for how safe they're making things, how much they're eliminating risk, and they're getting paid for how fast they can close the ticket and actually deliver that new feature. When it comes down to lowering risk or hurting our production flow production flow kind of wins out every time
Viktor 00:03:12.302 I must put myself in that bucket, man. I'm all in for being more secure as long as it's not m- more painful. If you tell me, "Oh, now you need to dance around your table for five minutes every time you need to do something I'm going to say no.
Dwayne 00:03:27.526 know what you're saying. You don't want to do the back flips to actually implement some kind of tooling above and beyond what you're already fighting with the mountain of tooling to make the thing work in the first place.
Viktor 00:03:38.645 I don't even mind additional tools. I'm more referring that, and I'm now speaking from the developer perspective, that I Like tools that are just doing their work in a way almost transparently, You give me a tool that will allow me to securely do something and that will just be happening in a way without much fuss I'm all for it
Dwayne 00:04:02.225 everybody wants that auto-magical, it just does the thing right. that was the great hope with AI, especially as we were marching toward the end of 2025, that, hey, the models are getting better. What if they just did all the security stuff for us? And turned out that wasn't a good idea after all either. So now we're in a world where we're producing more stuff than ever before faster than we've ever done it before, with more people touching code than literally ever before, and we still haven't addressed this elephant in the room, which is secrets are the bane of our existence. All this credentials and access management, all of this p- stuff powering all of our applications, all of our infrastructure. We just need to have that larger conversation. So I'm very happy to be here and having that conversation in public with you.
Darin 00:04:56.707 What are people doing right now for real?
Dwayne 00:04:59.577 Hard coding secrets is unfortunately the most common answer out there, it's not really a newcomer's fault in and of themselves. When most people are learn- learning to code Java, JavaScript, Python, what have you, it's rare they'll take an entire chunk of their life and go, "I'm gonna study security. I'm gonna go look at the world of cybersecurity, look at all the best practices, and implement that." No, everybody wants to build Hello World and then to do, and then whatever application they can put in production after that. And there's nothing wrong with that, but there's so much to security. There's so many things that you can get wrong that people get stuff wrong. That's the reality. there is another end to this. There's an extreme other end where teams are eliminating credentials altogether. They're moving to identity-based authorization and authentication patterns at scale. Some of the really cool stuff going on, like at Uber, they put out a paper a few months ago how they are doing agentic attestation hop to hop to sub-agent all through using open source the SPIRE framework or which is… No, SPIRE, which is the runtime for the SPIFFE framework Secure Production Identity Framework For Everyone. And they're doing that at scale, but that takes an engineering team that has resources and the political will to actually future-proof and start moving to these more scalable ways to handle authentication. The rest of the world is dealing with SaaS that they're still trying to tie together, and if you ask Claude Code or Cursor or any of the assistants right now, "Hey, put all these things together," it's gonna use whatever the template says, and so many templates right now say, "Oh, hard code your credential right here," and it works. And it does. Not safely, but it works
Viktor 00:06:48.358 One thing I would add is that Claude And, other agents will happily continue doing what you're doing, run them in a project in a, in a repo. You tell them, "Do this," and uh, if there is something similar already done, they will replicate that. Which is what I would do as a person, as a newcomer as well, "Oh, you want me to implement new functions? How are the other functions implemented? Like this. Cool. So that's how you do it here, right?" So o- once you start hard coding or doing silly stuff, you cannot expect agents to say "Uh, no, no, no, everything you did so far,
Dwayne 00:07:21.052 and I'm not here to bash AI. I'm actually writing two different talks around agentic security right now one on using hooks and the other on probability and understanding probabilistic systems. They're very interrelated. One's more of a how-to and the other is more of conceptual. But at the end of the day, they are trying to answer the prompt, and you're right, the way that makes you the happiest. But that goes back to what I said earlier. There's so many people that are coming into this world of natural language coding, vibe coding, whatever you wanna call it, that don't have that background or training to know like, "Hey, plain text credentials are actually really dangerous," or, "Hey, I shouldn't just always grab latest as soon as it becomes available automatically or automatically every time I run the agent." And there's just so many pitfalls to s- the security side of developing fast that it just makes it dangerous for everyone. Now, if you have that background, if you have that training, doing a quick code review, knowing what to look for, putting in deterministic tools at runtime to say, "Hey, let me check this against known anti-patterns. Let me check this against my tooling list," then we can go pretty far pretty fast and on paved paths, do it pretty safely. But again, most people aren't thinking in those terms 'cause it's hard to set up. What am I setting up? What tooling even exists? What am I looking out for? They don't wanna be security experts unless they're specifically going into the world of cybersecurity. They wanna get their product shipped
Viktor 00:08:52.180 I agree, but that's going back to my previous statement. Basically, if you're a developer, we don't expect you to be security expert, otherwise you would be security expert. So the question is, how do we make it secure without you even knowing it is secure, right? If it's agents, agents go try to talk to AWS and automagically somehow, whatever the solution is, they get the credentials in a secure way and I don't know about it. That's ideal in a way.
Dwayne 00:09:22.408 That's kind of the dream and always has been the dream. That's do it safely and for free. Fast, cheap, and now. Or fast, cheap, and good all at the same time
Viktor 00:09:33.368 I would not say for free. I feel that security is something that companies are probably willing to pay for before anything else. so I don't expect it, to be free. I don't think that many people do
Dwayne 00:09:43.003 this is a fun discussion I have a lot in the security communities. What do we mean by secure, and who are we talking to when we say secure? At what levels? 'Cause if we're talking about secure, there's security with a capital S where you can do everything exactly right, and wow, that is a giant set of rules, and it keeps growing. And if we look at just WAF rules web application firewall rules, as an example of rule sets that just keep growing wow. Millions of new rules available every day because that's how fast agents can write them now. But then there's security with kind of the smaller S of staying safe and secure. So then we start asking the question, what am I securing? What does security mean for me? What are my actual risks? And that's where the conversation starts changing from agents do it right or you're doing it right or wrong to I'm doing it with the least risk involved. And now for an individual, that's pretty hard unless, again, you have the security background. But for an enterprise, for the, a company of any size, there should be that collective will to we need to make sure that at least customer data is safe, at least transaction data is safe. And those are wider conversations than just, "Hey, developer, make sure your agent doesn't do it wrong." It's what access patterns even exist in the first place, And that's where, going back to the very, very beginning of this whole conversation, where vaults even come from. 'Cause developers locally have always been able to rely on, "You have to get to my machine first." enterprises are getting attacked constantly, I wish it was still true to say, "Yeah, if it's on your local, they'd have to get to your machine first." And what the last year has proven with info stealer attacks self-propagating worms like Shaihulud and Miasma and Chaindrop and all of the other crazy names out there, that now we're in a world where if it's reachable by you as a person in plain text, then it's probably gonna be compromised. And now we have to start talking about the world of capital S even at a local level for anything you don't want to be touched by, an attacker or even the local agent.
Darin 00:11:57.520 We've talked about credentials here at this point, but you just sort of leaned into it there. It's what can an agent connect to? And I'm thinking just connection strings. we can agree that hard coding credentials in an app is bad. That's easy. But we don't think twice about hard coding a connection string and then just replacing a couple of things, maybe. It seems like that would be almost worse than actually having the credential, because now I can have a script kitty going after my real endpoint if it leaks in some way, shape, or form. To me, that's just as bad as the credential
Dwayne 00:12:30.584 that actually is one of the stronger points of vaults. Like in theory, a vault is just a store of strings. Now there's more to it than that. There's, I don't wanna down talk HashiCorp Vault or any of the other providers out there, CyberArk, what have you. But at the end of the day, it's just string replacement at runtime through this third party that holds things encrypted at rest. You could do all your vars through a vault if you really wanted to, and that's goes back to that conversation about risk. Like what is risky for you? If it makes sense to hide your database connection strings that don't have a hard-coded credential in them, if they do have a hard-coded credential in them and you're using that string, that is a secret. That is a credential in and of itself. It starts getting to the finer distinction between a credential and a secret and a certificate. In my head, they're all the same thing. If they allow access, if they unlock or un- lock or unlock data, then we're talking about the same thing no matter what form and like finer semantic you wanna put on it
Darin 00:13:31.524 And the other way to think about this too is secrets managers don't guarantee that everything's gonna be okay because if you stuffed the password in there six years ago and you haven't touched it since then that password's probably somewhere else by now
Dwayne 00:13:43.758 There's a good chance. a good system well implemented, and again, I wanna give full grace to secret managers because I believe in them strongly uh, you can put rules in place. You can put governance policies in place to say, "This secret has not been touched by anything in a year. Maybe it shouldn't exist anymore." Send out some emails. Automate that process. That's pretty straightforward to do with internal hooks and security operations or even just plain old dev operation workflows within the tooling itself. It's just a matter of did we go in creating this access thinking about how it gets revoked when we did it? And that's the bigger cultural problem that we need to attack
Viktor 00:14:22.602 why do we have secrets in the first place? That's the part, puzzled me. Would, make more sense to just if, if you can confirm that I am me, I get access. And by I I mean person, agent, process, whatever. Would it make more sense to just authenticate the individual or a process instead of having a secret in the first place?
Dwayne 00:14:45.980 Secrets have been used for two main things uh, over the years, and it is a shortcut to get to authentication, which you were just talking about, and authorization, because you are bringing standing privilege with that key 99.99% of the time. You are logging in with that key as your identity. That means you're assuming a role inside of that system. And so many times, like GitHub keys are a great example. GitHub personal access tokens. You set the permissions allowed when you set the key, and anybody that has that key now has those permissions. So it's been a shortcut, really. I, I, I give a whole talk on this from pets to cattle to agents, on how we have seen this evolution. You can go find that on YouTube on a couple versions of it out there, about how we started with passwords weren't really meant to be a security idea. They were trying to tell who was logging into these really expensive machines at Berkeley. The fact that they were never secure to begin with if you go back and look at the database dumps from the early Berkeley systems, like Stephen Bourne, the guy who wrote SH, his password to Berkeley's Unix was bourne, lowercase. It's like hilarious. You go back and look at the people that wrote foundational computer science and their passwords weren't secure, because it was just meant to say, "Yeah, I actually logged in," or, "I logged in to play Zork all night and cost the school $300."
Darin 00:16:14.662 All right so I want to talk about identity here for a second, 'cause it's always interesting to me. We just sort of wave it off. It's like, oh yeah, we're doing identity knowledge, or, the system knows the identity. But at the end of the day, the identity is still going to be a credential. It may be short-lived, but it's a credential
Dwayne 00:16:31.253 Identity, I would argue, is represented by a credential that we accept, but the identity itself goes beyond that. And this is where you get into the philosophical argument of what identity means. this is where frameworks really come in handy. I'll always point to the Cloud Native Foundation SPIFFE. I mentioned it earlier, s- the Secure Production Identity Framework For Everyone, spiffe.io/book. You can read the book for free. It's 198 pages. It's changed your life. the idea that we need to persist identity over time, give it a full qualified path the same way you would with a URI, and then based on that, you can make things called identification documents, secure, verifiable identity documents. And now we're in the world of PKI, and now it's not just I'm asserting, it's I'm asserting through a system that isn't foolproof. There's actually just recent research on if you can get into root access and the controller pod, then well, you can issue your own certs, and you can start working and assuming identities. But that assumes domain controller-level access, which if someone's in your domain controller yeah, I don't think they're gonna attack SFits first. They might, but at that point, you own AD and all the human interactions
Viktor 00:17:48.257 Why don't we move when it comes to humans to kind of identity being my face or my fingerprints or both Isn't that really identity? like, I know that Darin is Darin because I recognize that person, There is no way you can put somebody else on the screen right now and I will say that's Darin
Dwayne 00:18:07.337 And now we're back to the Turing problem. It's the Chinese wall all over again. Just because he's responding in the way that the sounds you made… But now we're back into philosophy, and I don't wanna go down that path too deep. I agree with you. We should be using things that are permanently identifiable and unique to us over time. I think we'll get to a world of DNA unlocking YubiKeys at certain point in the future. the people that believe in retina scans for this. at what level is the juice worth the squeeze when it comes to the security? For me personally, the, one of the reasons I'm on a MacBook Pro right now, and one of the reasons I personally like it is I just touch the thing with my finger and it unlocks. Same thing with my Android phone. Same thing with the YubiKey with bio that I have that I unlock my Linux box with. But I believe in biometrics, and that's awesome. The problem is technology's not ubiquitous and free. Who was it? "The future's already here, but it's just not evenly distributed"? William Gibson
Viktor 00:19:04.230 for example, when we talk about developers specifically, majority of them has Mac, majority of them has that button, as far as I know, we use it exclusively for me to approve some purchase on Amazon, not for my work. At least, maybe I'm wrong, so correct me. I don't recall anybody ever asking me for my fingerprint or face scan when entering a system, and the technology's there. I have it
Dwayne 00:19:32.942 the people at FIDO, or the FIDO2 Alliance absolutely 1,000% agree with you, and we wish it was higher than like, 18% adoption for passkeys, 'cause that's the general broad term for what you're describing is a passkey. and we should be adopting them faster. We made the classic mistake we made with OAuth that OAuth is not a way to do a thing. It's a description of a way to do a thing. The actual implementation of it takes 1,000 different forms, m- at least 10,000 different forms. uh, The old joke with Active Directory. Active Directory is not one technology. It's 100,000 implementations of an idea of technology per company. The future is just not evenly distributed. I actually don't believe that the majority of developers are on w- Macs. I just don't. Linux boxes, cheap Linux boxes ThinkPads from 10 years ago. Thinking pure global perspective yeah, if you're a student coming up right now and you're vibe coding for the first time, you have just enough juice on your machine to run a coding agent, and that doesn't take that much anymore 'cause you're doing a lot of processing in the cloud now. the access to cheaper doesn't equal the access to the experience you're, I think we're having
Darin 00:20:43.738 So if AI services have become or are in the process of becoming a credential class of all their own, what should it look like? 'Cause going back to what Viktor was just saying about me, it's like he can see me and he knows that me- that it's me. He can see that I ha- I'm holding up five fingers right now, You've seen all the, the silly AI interactions with people trying to do an interview with an AI bot. It's always pretty funny. They hold up 10 fingers on one hand. But that goes away with AI services, We no longer have biometrics until Skynet with the AI stuff. what is the right way to do that?
Dwayne 00:21:20.654 there's two different issues there. One is, are you trying to attest for the agent's actions itself and say this entity can or cannot be granted authorization within a system? And if it's that, then we're just talking about workloads. We're just talking about SPIFFE again. I firmly believe there is a divide in the identity community that some people feel that blended identities or digital twins represent a different kind of approach that is needed. I disagree. I'll say that on the record. If it's a running piece of software executing a math problem, that's a workload. We already know how to identify those. The benefit of workloads is we can be there when they're created. We can attest to cryptographical truths that this was born on this process, on this chip, on this piece of hardware, with this firmware, at this time. It was expected, it was spawned by this process. All the signatures match down the line, and every five minutes, I'm going to issue a new SVID that says, "Yep, this is still alive, this is still alive, this is still alive, this is still alive." And if anybody asks me, the issuer, I will look at it and say, "Yeah, that key matches. We're good. That's definitely provably that thing." that's not science fiction. That's an eight-year-old technology. It's free and open source, called SPIRE. There's a lot of companies selling this awesome technology at scale, and they would love to talk to you. I don't wanna really name drop a bunch of them on here 'cause I don't work for them, but there are many awesome ones, and the one I will talk about publicly and not afraid to make anybody else mad, is AWS. AWS has had STS, Security Token Service, for a long time. Is it the most widely used thing? No, not yet, but it should be. But it was bound into just the AWS infrastructure for so long, until the end of last year.
And One Of The More Exciting Things They Did At Re 00:23:12.513 Invent, in my opinion, was they publicly announced federation for it, which means you can take that same token that STS is saying, "Yes, I can prove this thing is this thing, this entity, this workload, this call is this thing," and you can take it to another service like Azure or GCP or Databricks or there's a lot of services out there that will do this, and basically using OIDC, it makes the call back and says, "Hey, can we prove this identity is this identity?" And once we prove the identity is the identity, then we have the other problem to deal with, which is the authorization side. But now we're talking about roles, and that world's evolving really quick right now. Pointing back to AWS again, they have introduced Dogwood, which is specifically how do you give roles and handle authorization for agents when an agent's making a call. It's like bleeding edge right now. I'm not super knowledgeable on it. I'm trying to write a paper on it, but eh, trying to learn. ' Viktor: one thing that is confusing me when it comes to agents, right? In the past, I, would say, okay, so you, you're using an agent right now. That agent is somehow authenticated, identified, we know what it is. It talks to some service, it can talk or it cannot easy, right? But now we are spinning agents like crazy. What is the identity of agents in plural? Every process running on that machine It cannot be every process started by me because I'm not even starting majority of the agents anymore, right? They're just floating around
Dwayne 00:24:46.760 I'll push back on that a little bit. It's not every process running on your machine unless every process in your machine is involved in the harness and workflow of that LLM call. So we can bound it by what is an agentic call doing? If there's no agent called, if it's just an agent calling a tool we're still bound by an agent calling a tool. So it's easier on a whiteboard than it is in conversation, and that's kind of one of the complicated bits about it. But the actual architecture for any agentic system looks shockingly amount like web architecture from 20 years ago. It's just the only thing that we don't understand is that black box LLM call that matches a bunch of factorization? that get you to a guess of a next vector based on tokens. The math gets complicated really quick. It's like very, very long problems, but the actual… This call gets made and then something else happens if the if-then statement between systems, that's just a call. One of the beauties of OpenClaw, why it worked and why it so spread so fast and what's such a beautiful architecture is there's just a simple hub router in the middle, and everything goes back through the hub. So where do you put the controls? Where can you put the gates? Where can you say what's calling what? There's one central point that's pretty easy to point to and say, "Hey, that is where the calls are happening. But who's making the call? That goes back to the identity side of it. So which agent got spun up by what? That's an interesting question. What agents can do what? Oh, that's something we can actually address at a gating level, and that gets us back to stopping agents from acting weird and with our credentials
Darin 00:26:34.864 I think where he was going, at least this is the way I heard it, was I've initiated a workflow with my agent, that's that. So that makes sense. It, whatever work that agent does should be on my behalf. Digital twin-ish, right? But any of the agents it spins up, is it still me? Is it not me? Does it have to be me? That's the problem. It's, It's that last question, does it have to be me? Because it may not make sense
Viktor 00:27:04.434 just to continue, if any agent spun up on that machine carrying that same identity, that means that literally anybody who can spin up agents on that machine gets th- that identity
Dwayne 00:27:14.891 there's a CSA framework around AI authority. I forget the name of their framework off the top of my head. I have to go look it up. But it talks about this in depth. So the CSA has actually got a framework. There's a few other frameworks people are, have been talking about. But on behalf of, and that is an interesting question, but it points back to the philosophical argument of " I and identity. This goes back to Hofstadter's I'm a Strange Loop. like where does that, arise? That, that's an interesting conversation. I'm not saying it's not, but from a practical perspective, it's what authorization grants were granted to what identities at what point in time? You as a human granted a ag- an agent a authorization grant based on its identity at the time. It spins up another agent, and it passes that same authorization grant to the new identity if it's allowed to do so. we can track over time. That's the attestation I was talking about that Uber is working on
Viktor 00:28:15.639 Just to be clear I'm referring to my agent executing a shell command that spins up an agent, then that agent It spins up through a shell command would get the same identity the agent in question got in the first place?
Dwayne 00:28:30.159 It depends. on how your setup is set up. if, you were giving the agent your GitHub access token, and it can give it to any other agents it spins up, then yes, they're all gonna act with your authority on that particular system. However, if you say, "Here's the allow list we've put into the OICD setup we did, and only these known agents can interact with that," and another agent shows up, and it says, "I wanna do this thing that wasn't on the contextual allow list," th-there's a framework coming out called AuthZen it's gonna be baked into OAuth, that literally adds a field called context inside of the JSON for authorization calls to say, " That's not supposed to be here. You're not supposed to be here, and you're not supposed to do that." The answer is no. Now, can you 100% deterministically say it will only happen in that flow? No. Can it try to spin up other agents? Can it try to invoke other things? That's the probabilistic nature we're dealing with. So it's, it's fun conversation, but for pure security, the attestation chain of this authority was granted at this point in time, and here's where that authority flowed from, should go back to a human, and it does go back to a human. But it's traceable. This is provable, like you can just prove this if you have the technology set up to say, "Here's the chain. Here's exactly what happened."
Darin 00:30:00.061 You're making more and more arguments that we should not be using AI in business
Dwayne 00:30:03.561 No I disagree. I use AI in business all the time
Viktor 00:30:06.655 Darin I still trust agents more than I trust humans They're more reasonable than people are
Dwayne 00:30:11.916 I have yet to meet an outright malicious agent. I'll put it that way
Viktor 00:30:16.673 There we go
Darin 00:30:17.969 I'm just thinking how do we make this even make sense? Because to me, if I have a great agent harness set up, I'm handing off work between different agents that have already their own identity and authorizations, but yet it's my fault because I called them, right? If something goes sideways, it's not their fault, it's actually my fault for calling them
Viktor 00:30:37.778 That's how good manager should act I assembled this team messed it up, it's my fault." 99% of the managers do not react like that but that's how they should be reacting. Kinda you don't take only the glory man, you take the fault as well
Dwayne 00:30:52.940 we share glory and responsibility is mine. I like that model, and that's the way good leaders are. Again, I can't say, "Hey, developers don't wanna do this right. Agent coding users don't wanna do it right." But wow, we have all been overwhelmed in the prompt to say, "Hey allow," and eventually we just go YOLO. Eventually we just hit allow and we don't even know, read it anymore. That fatigue is real. And yeah, that's the other argument for access should never be standing. Access should be temporarily granted, that's a tuning exercise in the harness. That's a tuning exercise in how we grant authority. If everybody was using X.509 certs for every kind of grant, or we move to a world of JWTs, JSON Web Tokens, that have expirations no longer than three minutes out or even 90 seconds out there's no reason we can't do it. Technically, it is feasible. Just it's much easier to grant something that, "Oh well, it's gonna need to work next week, so let's just let it run forever."
Darin 00:31:52.433 You were leaning in there for a second. You said, 90 seconds, three minutes. That's just way too long for me. Five seconds, but I can't expect my issuer to actually turn stuff around that fast before it expires
Viktor 00:32:04.874 Okay. Darin, create an EKS cluster in AWS in how many
Darin 00:32:09.207 let's see. That would be 1,800 seconds, I believe.
Viktor 00:32:11.778 Yeah, try to do it and tell me how It went
Darin 00:32:15.053 that's the problem is we wanna get, want it as small as possible, but then we run into this whole thing called physics to where how do we get back something fast enough from a centralized issuer? And that's another problem to me is it's still air-- I'm air quoting "centralized." Sure, it's HA8 and everything else, but if that's down, then everything's down. That's the argument for against using any kind of issuer, in my opinion
Dwayne 00:32:42.485 Yeah. that is the classic argument against, PKI. too slow
Darin 00:32:46.495 Too slow, and if it's down, I'm done
Dwayne 00:32:48.927 Yeah. But if that piece of your critical infrastructure's down, is it worth exposing all your PII to the internet without any security in it? Or do we not do business? That's the hard challenge of risk, I am not naive enough to ever say security risks should rule all. It's the only thing we should think about. But that's why we have board of directors. That's why we have CISOs that, talk to audit committees, that what is the right trade-off there? I got into a wonderful back and forth from stage once with somebody pretty prominent in the PHP community who made the exact same argument that said, "Look, I'm making 40 million transactions a day happen. You add 10 extra milliseconds on top of my calls, you do the math. 10 times 40 million is not a little amount of milliseconds. That adds up quick. You have introduced latency that's unacceptable in my test rig." that's why I said there, at some point, there has to be an acceptable risk level that you can tolerate And that's just physics
Darin 00:33:53.506 Their problem is they were using PHP instead of being in assembly. that's their real problem
Viktor 00:33:58.307 I was listening and, and my brain had trouble processing that somebody using PHP is concerned about latency that's like me saying I'm walking from Barcelona to St. Petersburg And I'm concerned that I'm not gonna get there fast enough
Dwayne 00:34:14.551 2026 will finally be the year where the internet stops using PHP for the majority of its stuff, so
Darin 00:34:20.156 Wow, that's also the year of the Linux desktop, so here we go So what happens when we do have a failure? Like all of a sudden our credential leaks, our connection string leaks. Usually it's a fire drill, What should be our reaction now? I mean, It's 2026. We should have a better reaction than we did in 1990
Dwayne 00:34:39.854 The answer is still rotation, and that goes back to vaults. kind of an ideal situation, what goes on every single day I haven't even talked about who I work for or what my company does for a living, but this is something our customers face day in, day out. We have a properly vaulted secret. It is being used correctly across our CI/CD pipeline, and now for some reason, somebody debugging something, somebody that didn't know what they were doing, has copy-pasted this into Confluence, and now Confluence is also being compromised because someone leaked the key to Confluence. I, I've taken many red teaming classes in my life, and almost every single time it's like, "I'm gonna teach you how to break into Jenkins and dump all the credentials in Jenkins." That's modern hacking, is just get to the credentials as fast as possible. But yeah what does that actually allow access to, and how fast can we rotate it without breaking anything? Well, If the case of it's already accounted for in a vault call, just rotate it in the vault. There should be some blue-green deployment strategy going on somewhere if you've architected correctly, and nothing ever goes down. It just auto-magically falls over and the risk disappears because you knew early enough that, ah, this got leaked into Confluence, but no one's attacked Confluence yet, and we have that window. But the sad reality is that's human speed. Everything I just described describes a human speed situation. Breakout times, according to CrowdStrike now, are dwindling down to there was one attack they saw from the time of the attack actually broke through to the time exfiltration started was 27 seconds. How did it do that? How did it do the lateral movement that fast and unlock and start exfiltrating that fast? It harvested all the credentials it could. That's happening at machine speed. This is what drives all these stupid supply chain attacks, is how fast can we dump your creds? How fast can we publish forward, and how fast can we exfiltrate anything that's attached to those systems? And then self-propagate along the way
Darin 00:36:40.538 Tell us a little bit about what your company does, GitGuardian, because, you know, it sounds like you're trying to guard Git. That's what it sounds like. Is it in the name?
Dwayne 00:36:50.391 that was how we originally, originally started. Our founders back in… They were doing research in 2017, but in 2018 one of the things they noticed was, "Hey, we keep running into these hard-coded credentials out on public GitHub." And they started calling people and writing emails to the committers and saying, "Hey, you did this in public," just to see what would happen. And some people got really mad at them. Some people ignored them. Most people ignored them. Some people came back and said, "Hey, thanks. Could you do that for other stuff, like find secrets?" And that's literally how the company started. Since then we've been building detectors. We still scan every new secret or every new commit that hits GitHub in public. That's our public monitoring. And then we do identity stitching on the back side to say, "Hey, somebody with a Yahoo email address is sending a bunch of working Terraform and AWS platform or infrastructure secrets to a repo called Private CISA," and they've been doing this for months. And our security researcher finally like, started looking at the context of it and realizing like, oh, the URLs that are paired with these credentials go to working infrastructure inside of CISA. So that's part of what we do. We do the research to say, "Hey, these secrets over here belong to this company over here." We call that our Good Samaritan program where we try to do security disclosures, reach out to security teams and say, "Hey, this is happening in public. Don't do that." But where we work as a company is that same powerful engine, we can hook it up to anything under the sun if it's got text. So it started with Git and Git repos. GitGuardian was the natural name of the company in 2018. But it's so far beyond that. The platform ticketing systems your messaging systems, your package registries, your container registries CI/CD pipelines themselves. Now we've started looking at the developer laptop through some um, and MDM tooling. So we have a really powerful CLI. So now we can install that through Jamf or Intune or IRU or whatever MDM tools you like and say, "Hey, we're gonna give all your developers these ultra powerful tools. At the same time, we're also going to see what's on that machine and give you one true clear view across your entire credential layer, everywhere credentials could exist." And that also includes the vaults. That's also where secrets are supposed to be. We don't copy them out of the vault. We take a fingerprint of a hash of a secret and then all the metadata and build all this contextual map for you to give you here's your one true visualization, all the context, all of your secrets, no matter where they live And then help teams figure out how to assign risk to that. We have some automations around that, but our goal is to detect secrets wherever they exist, help you remediate that situation, and then give developers tools to prevent that. That's what goes back to our CLI. And that's stuff I'm living in day in, day out, working in our AI hooks pre-commit hooks, and other cool stuff we got brewing on the locals
Darin 00:39:52.778 So if you want to find out more about GitGuardian, you can find it at gitguardian.com. That's Git, G-I-T, Guardian, the way you'd spell Guardian, but I'll spell it anyway, G-U-A-R-D-I-A-N.com. And all of Dwayne's contact information will be down in the episode description. Dwayne, thanks for hanging out today
Dwayne 00:40:12.268 Oh man, thanks so much for having me. It was great to be here. I appreciate it We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox.com/contact. If you subscribe through Apple Podcast, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox