David
00:00:00.000
So for example, when you-- like you start, you said at the beginning you install a package with npm install, and that's your first mistake because that allows all sorts of arbitrary code execution on your system, There's a load of different scripts in there. There's a post-install, there's a pre-install, and npm allows the developers just to run arbitrary code on your laptop before and after a package is installed. And that's just one of the vectors that are used to get malware onto developer laptops or CI flows or even into production. This is DevOps Paradox, episode number 366, How to Prevent NPM Supply Chain Attacks
Darin
00:01:32.361
Viktor, you and I are going to be talking about a language today that we probably don't want to talk about at all
Darin
00:01:41.861
maybe, Maybe not. How about JavaScript? Let me set up the scenario for you. Everybody that's dealt with any kind of JavaScript, TypeScript probably has run NPM install or some variation of that in the past 24 hours, statistics say anything at least one of those items you pulled down was probably compromised this week, and you just don't know it yet
Viktor
00:02:06.780
Most likely. Especially NPM. You down-- When you download dependencies, you download half internet. One of them must be problematic
Darin
00:02:15.000
On today's show, we have David Mytton on with us to talk about the state of the chaos that is NPM. David, how are you doing?
Darin
00:02:27.092
Now, David is with Arcjet, and since we haven't talked about this, David, I want to go ahead and just draw the comparisons because I'm a Java guy. So Sonatype runs Maven Central. The way I've sort of read it is Arcjet runs the npm registry. Is that correct?
David
00:02:44.948
We exist inside JavaScript applications, so developers will use Archit to help protect their applications, but we don't have anything to do with npm other than our packages are distributed on npm, as npm is part of GitHub and Microsoft these days.
Darin
00:03:01.088
Yeah. I was doing some research and I was like, " Wait, npm was bought by Microsoft?"
Darin
00:03:15.905
I know. I know. It was just odd. There was another variation. You would think since Microsoft owns npm that… Or I say Microsoft, GitHub, Microsoft. You would think all these problems would already be solved
David
00:03:31.575
You would think, but it seems like npm has just been forgotten for years, and it's only really as we're seeing all of these malware distributions, worms, whatever you wanna call them, suddenly appearing to exploit the insecurity of npm that it has started to get attention. And it's taken a while for the GitHub or Microsoft or whoever is controlling it, it's taken a while for them to actually do anything in response to the issues we've seen recently.
Viktor
00:04:03.415
I'm curious, when we talk about, security of packages of one language or another, usually the focus is on, hey, how I ensure that on the registry level those packages are secured, scanned, what so not, right? I understand since you're not a registry, you have a different approach or
David
00:04:22.725
Yeah we, we sit alongside the usage of the registry. So JavaScript developers have all sorts of security concerns, and the supply chain is just a part of that. So we sit inside the application. We help with things once the application goes into production, so that's dealing with what AI agents are doing, giving them authentication, authorization tool calls. But we're a big user of the npm registry, and we sit inside the JavaScript ecosystem, and so we've had to take a lot of precautions to protect our packages because if, as we're seeing, packages are being compromised left and center, if our packages get compromised, then that affects all of our customers, and that's no good if you're a security tool.
David
00:05:08.655
I think this goes back to the original days of the internet when everything was trusted, and npm came up in that era where it was really just of convenience to distribute packages, and no one was using it as a, as an attack vector. And that's just been the philosophy of whether it's npm or PyPI in the Python ecosystem less so on the Go side of things and the kind of older languages. But they were there just as a community resource, and that's just been abused over time. So for example, when you-- like you start, you said at the beginning you install a package with npm install, and that's your first mistake because that allows all sorts of arbitrary code execution on your system, There's a load of different scripts in there. There's a post-install, there's a pre-install, and npm allows the developers just to run arbitrary code on your laptop before and after a package is installed. And that's just one of the vectors that are used to get malware onto developer laptops or CI flows or even into production. So defense in depth means looking at all these different layers and seeing where things can come in.
Viktor
00:06:17.336
That feels like architectural mistakes at least in retrospective, that should be solved at architectural level in a way, right?
David
00:06:28.262
Yeah, and it's a mistake when you look at it in retrospective, but in, if y- at the past, it's kind of just a naivety of this would be convenient. The pre-install hook is very convenient 'cause you might wanna compile something while you're doing the package installation for the local architecture of that laptop. It's very reasonable. But when you actually think about it that also gives you arbitrary code execution on the laptop or wherever it's being installed, which, like I said, in retrospect, is probably not a good idea. Certainly not automated without confirmation or being able to easily inspect the contents of what's being executed.
Viktor
00:07:02.472
But I, I'm not deep into the ecosystem myself just a tourist in a way. Is anybody trying to fix that at that level? Okay, maybe actually we should stop doing this in the first place
David
00:07:15.552
Not in terms of the post and pre-install hook side of things. Not in terms of npm or the tooling there. There are configuration options to disable that, but they're not the default. And so we've seen an explosion of vendors that will wrap your installation or provide a firewall or scanning before those before the script actually gets onto the developer laptop, it's not solving the root cause. It's a treatment of the symptom rather than trying to s- actually solve the underlying problem, which would be to lock them down and disable them by default. And then maybe if a developer does want to do that, then you could ask for permission and explain what's gonna happen. But at the current state of things, it's, it happens by default, and so this becomes an easy vector for someone to install malware and to ex-exfiltrate things from your, from your environment and then just clean itself up and delete itself as part of the post-install hook.
Viktor
00:08:11.882
But it also feels like an easy fix on NPM side, right? Kind of, Okay let, let's just not make it default
David
00:08:19.452
Not make it default. That would be a, a good idea, and maybe that's the direction it will go in. But this goes back to our earlier point of that npm has languished for a while, almost in maintenance mode with a skeleton crew maintaining it. And it's only in, in the last few months that we started to see real investment in changes to npm as a registry and also the tooling itself
Darin
00:08:45.532
With the registry, they added in the trusted publishing, which Viktor really hates. We talked about that a few days ago. And signings and everything else, I mean, in theory, those are good things, but are they putting in too much friction to actually get anything done?
David
00:09:01.672
I'm curious what Viktor thinks this, but I think it's necessary friction, because you are rarely releasing packages, it adds guards in place. So you-- there's a few different layers there. You've got trusted publishing, so when you publish something to the registry, ideally from a GitHub action or another build workflow that is public, it will connect to the registry using OIDC rather than a long-lived token. So the connection is ephemeral. And then it can point back to the provenance of the build so that you can prove that a package that has been released came from a particular build. That then relies on things like immutable releases on GitHub, so you can actually not just-- Currently, without that, you can just delete the release. But with immutable releases, they can't be deleted. And then what has been added most recently is staged releases, whereas previously you would just run a single command and it would release the package out to npm. Now you need to have a release and then an approve step, which you could already do on GitHub by configuring the release pipeline there. But it's now built into the npm registry, which allows you to require two people to publish, which in theory should mitigate the idea of a single maintainer being breached. But I'm curious why you don't think that's a good i- or what your
Viktor
00:10:23.667
No, uh, it was more that I wasn't aware of until now that you can do OIDC authentication through uh, GitHub actions, 'cause every 90 days I end up with failed builds trying to figure out what's wrong and then go and fix that repo only to find out that another repo, and I have a bunch of repos right, that they're all failing for the same reason, right? And especially now that basically I hi- I hid that on a date because gone are the days when we release something every three months, uh, I release multiple times a day
David
00:10:58.098
yeah, that's right. And you want those short-lived tokens because they get stolen, you don't know that they need to be revoked. And OIDC, it's not gonna solve every security problem, but it does give you a traceable, very short-lived token. ideally, you're then making sure that well, you trust GitHub, hopefully, to run that, minting process securely rather than having to run it yourself and then it getting popped from your environment or whatever. But I think the problem-- the criticism of npm generally and then GitHub and Microsoft as custodians of this is they've been quite slow, and there's been a resistance within the community to adding some of these things, like two-factor authentication to popular maintainers
Viktor
00:11:39.330
Is it one of the things that Microsoft actually cares about? Or is it one of, an- another one of those purchases like, why not? It's there
David
00:11:47.724
it was GitHub that bought npm, and then Microsoft bought GitHub, and I suppose they've come along for the ride, and it's not really been prioritized. And compar- compared to what revenue you can get from AI, obviously npm is not gonna generate the token-based revenue. But it is a load-bearing piece of critical infrastructure, and it could really do with proper attention
David
00:12:14.271
Absolutely. Yeah. There's so much dependent on both npm and PyPI, and I think they've s- they've suddenly started to see that load and that stress from all the packages, with JavaScript really being in the front end and the applications that people are building and the workflows, and then Python with all the data science and training the models.
Darin
00:12:34.511
If you're running the registry team right now, what would be one thing you would ship next week?
David
00:12:38.541
They just shipped the main thing that I would have improved, which is the staged releases requiring two people to really get a package out. It's still opt-in, and it is a breaking change if you've scripted your workflows to do the package releases through CI. That would be the main thing because then it mitigates a single point of failure with a maintainer losing credentials or a token or whatever it is. But I think forcing that, and they did do that with old tokens where they just ma- invalidated them all. And that came after the, I think it was the Shai Hulud worm from last year. But that is a big step but it assumes you've got multiple people involved in a project, and that does add friction.
Viktor
00:13:20.478
wait, you earlier said that mitigates the problem of when if a person use-- loses a token. Are we saying that if I lose a token for my pet projects, there is no way to recuperate it somehow?
David
00:13:35.558
I mean in terms of if you mint a long-lived token that has the permissions to do the publishing, and you can now disable that. And even with that token, you can require someone else to approve the publish step. And so the idea is you have to have two approvers, even if one of those gets compromised, so as it makes it less likely that the other one's gonna get compromised.
Darin
00:14:00.484
I'm thinking about that staged scenario. You're saying it's two. So basically the way I hear it is two people would build the exact same thing, push it to the registry, then we would get a checkbox. Is that the way it works
David
00:14:11.624
I think it's one person does the build and pushes it and approves the release, and then a second person has to also approve the release. They're not both building it, so I suppose you can then have the CI still run the build and then have someone else trigger the approval. Now this just protects the release part, and it doesn't protect the release of packages containing malware or other scripts. That's an entirely separate problem that still exists.
Darin
00:14:38.974
I'd have to say, okay, two is fine, but anybody that's worked in distributed systems know that two means split brain. We need three to get to quorum. But who knows? That's-- what do I know? I'm not a JavaScript person. Pnpm was supposed to help with some of this stuff, right? Wasn't that supposed to be magic?
David
00:14:55.816
pnpm is a package manager, just approaches things in a different way, but it still uses the npm registry. So npm is the, is a CLI uh, it's a package manager. It's the default in the JavaScript community. And for the CLI, like npm install package X you can do that with e- with pnpm. But pnpm still reads from the npm registry and still have the same problem. And the only alternatives there's another package registry by the team at Deno called JSR, and they tried to solve some of these problems from the beginning, but it just didn't it didn't gain in popularity. And npm as a registry has so much momentum and gravity that it still remains the default
Darin
00:15:40.574
It's just sad. I'm sorry. It's just sad 'cause now you have a single point of failure
Viktor
00:15:50.282
no, everywhere in terms that if you have a registry or something else or a place from where people get things that has a lot of gravity it's close to impossible to replace it, For any competition to, "Hey, we are very successful. We reached one 0.01% users," or packages or whatever it is. That's close to impossible. Similar to like when many people moved from Blue Sk- from X to Blue Sky and now they're all back to X simply because everybody's there.
David
00:16:21.817
Yeah, it's the network effects side of things, and JS R had a, had a good go at it. It's still, still there, still exists. But it's just not used, and it's-- every- everyone goes to the NPM registry, and really most language ecosystems just have one way of doing things, even if there exist other things around the side just because that's the default and it's what e-everyone knows you can get everything from that one place
Darin
00:16:46.307
Talk about npm ci, right? 'Cause in theory, that's what we're… That's the bundling. Is that the way you bundle things? Again, I'm not a JavaScript, TypeScript person
David
00:16:53.807
Yes, that's right. So npm ci is the correct way to install packages that you've already installed. So you can't use npm ci until you have a lock file, and you create the lock file when you, when you run npm install, and that solves some of the problems. So when you then need to reinstall the same packages, whether it's locally or in CI then you use npm ci, and it uses that lock file to ensure the exact same things are installed. But one of the challenges with again, going back to the defaults is when you install a new package with npm install, it will install it with a version range rather than a pinned version. And this causes a lot of problems down the dependency tree because other packages may depend on a package that is compromised, and they will just automatically bring in that compromised version if you run npm install as part of your normal process.
Viktor
00:17:50.723
Th- there are many other issues with, npm or JavaScript in general yet excluding frontend now simply because there is no choice. People persist with it. Why?
Viktor
00:18:05.665
Yeah. I'm not, and I'm, and excluding front-end because obviously in front-end you're gonna do JavaScript, right? is is it some level of masochism or simply what I'm familiar with or
David
00:18:16.928
Yeah, this is a good question. And I didn't start with JavaScript. I came up in first in PHP, but then very quickly over to Python and then Go, and have more recently started writing JavaScript on the back end. And it feels very strange to me to write back end code in a language that was designed for browsers, essentially. And TypeScript has solved a lot of the pain by, adding types, which are quite helpful, I feel. But it does feel strange when you're building an API to write it in JavaScript. But that's because I've been around writing code for almost twenty years, and I understand the difference between front end and back end, and it was always a traditional split. They were two different skill sets. You would work on very different things, very different understanding of how systems work, and often still engineering. It's not that you don't have to do a lot of difficult things with JavaScript in many complex applications. But if I was building an API, personally, I would use Go because that's what it is written for, and there are other languages for different problems. But there is a convenience, and I think that's the key thing with developers and humans in general, is there's a convenience in writing everything in a single language. Like your front end goes in TypeScript and your back end APIs go in TypeScript. And just makes it a lot easier with the build pipeline and where you're deploying it to and managing the runtime and the dependencies. But it gives you a lot of opportunity to shoot yourself in the foot
Viktor
00:19:43.733
I'm especially curious because there is vast majority of agents today are TypeScript, if I'm not mistaken, and, it's a binary that you distribute to everybody else. It feels almost unnatural to choose Instead of you just mentioned Go, right? I can lit- I, it's a bi- I can compile it to binary. You get the binary, done. it feels so forced in a way
David
00:20:13.588
Yeah. Again, convenience. And I think I, I would like to blame Electron for a lot of this because it allows you to package your application inside a browser, you get hundreds and hundreds of megabytes of dependencies just because everything's actually just a web browser to build a desktop app when actually it could be a native binary. again, it goes back to the convenience of writing everything in TypeScript and then deploying it to the web and having the code all coupled. Just if you're building a simple application, it does-- it is nice. It's certainly nice to be able to just have a single code base. But as soon as it becomes more complicated, then I think you start to hit up against a lot of the pains
Darin
00:20:51.718
Getting to that point is, am I creating a production application or not, or is it a toy? And that's how much effort I want to put in
David
00:20:59.028
Definitely. this is potentially a historical problem. You'd write it as a prototype, and then that would suddenly become a production application, and you'd have to support it. And rewriting, I don't think there's ever a good idea to rewrite a production application from scratch. You'd have to componentize it and then extract the bits. But now maybe with AI, that can solve it, and you can just say to Claude, "Rewrite this JavaScript app and split it up into proper languages for each of the components."
Viktor
00:21:24.718
Yeah, but you know, Claude and other agents, they also increase the probability of… Because everybody's working now almost daily or weekly on some pet project, and then one out of 100 become a production application, and that's happening every single, every second week or something like that, right?
David
00:21:44.846
Yeah, definitely. And it's even more of a problem in the startup world because you're so time constrained. You have a certain amount of money, a certain amount of runway left. You're trying to iterate as, as quickly as you possibly can on the idea to see if it's gonna work or not. And then suddenly, when you have something that's very successful, you have now a lot of tech debt to maintain, as opposed to maybe being at a larger organization that is incredibly popular and, and profitable. You maybe have the time and luxury of, designing it properly.
Darin
00:22:13.966
I want to go back to the tech debt and designing properly. We sort of mentioned it before talking about pinning. Shouldn't everything be pinned from the beginning? This whole fuzzy range stuff is for the birds. Again, I'm from Java world, so it doesn't make any sense to me
David
00:22:30.351
I agree, and that's the default I set all my projects to. You can put it into the… There's an npm config file, and you can make it pin all your dependencies. it doesn't solve the problems, but it, mitigates a lot of the issues of suddenly all your dependencies have updated to a range you weren't expecting. That's a security issue, but I also think it-- but stuff could break, right? Not everyone follows semantic versioning, and you'll get a minor point release that actually changes things you weren't expecting. But the main downside of doing that is when you have conflicts across the dependency tree where one package needs a newer version or even an older version, and you're pinned on one, and you then get dependency conflict. I don't know if you ever remember trying to install the old RPM packages on Linux, and then you'd have a dependency that was depending on something else, and you'd have to do a manual resolution and all the different… That, that was a nightmare world, and some package managers are not good at resolving and isolating dependencies that have mismatched versions. I think that's one of the things that pnpm does solve quite nicely, is how it isolates the dependencies. But yeah, I think it should be the default for every language really.
Darin
00:23:41.647
Let's say something does escape into your environment and you've been hit, if you will. How are you gonna even notice it, especially if it's already uninstalled itself after installing itself?
David
00:23:52.953
Yeah, this is where defense in depth comes in. So what we've been doing at Archjet until very recently is having dev containers for our dev environment, where e-everything is installed on the developer's laptop inside a container. There's absolutely nothing on the host. So when you install dependencies, if there happened to be something in the dependency tree that installed malware, it would be contained within the container and have no access to keys on disk, or pulling stuff out of your password manager, or stealing cookies from your browser store. And that's a good way to basically just isolate the dev environment. We've moved away from dev containers because we are often now running multiple work trees in parallel just because we're using agents and AI, and instead we use VMs, but it's the same. It-- where VM is even stronger guarantee of isolation. But it's the same principle is that each dev environment is entirely isolated. And then on the host, we have MDM installed. We use Fleet and SentinelOne, which does scanning of these kind of indicators of compromise and common malware. But if you don't have one of those tools installed, then maybe the first you'll hear about it is when someone accesses your production environment or you get logged out of all of your sessions of something because someone's taken your tokens, and it's pretty difficult to detect. You could have a canary token, which triggers an alert if it gets used. Uh, It's just sitting there on disk. That's something that is, can be quite effective. But it is very difficult to detect these things outside of having your normal logging and monitoring infrastructure up and running on production and all of your developer laptops.
David
00:25:39.544
Yeah, so all of our team use macOS, and we use a tool called OpStack, which bundles containers. It's basically a Docker replacement, but also has a ability to run VMs, and they're native VMs on, on macOS. And I'm not entirely sure which framework it uses, but the later versions of macOS have a built-in virtualization API that allows you to create these micro VMs. They spin up in, in just a couple of seconds. We install all our dependencies. It's very easy to duplicate the environment, so we can have multiple work trees. But it does give the developer the i- an isolated environment. And something that I've been seeing in the community more is the is more cloud dev environments. I think this was a thing a few years ago, and GitHub tried to do codespaces, which to my knowledge didn't really go anywhere, even though it was a really cool concept. But the developer laptop was just so powerful, it didn't make sense to pay a lot of money to have VMs running in the cloud with your dev environment. But now the workflow with AI is increasingly having multiple dev environments running in parallel. I think this is something that's gonna come back starting on the developer laptop still, but I think we're gonna see more dev VMs being used
Darin
00:26:51.137
I'm trying to get my head around, because I'm not using VMs that way. I'm very familiar with them, but not the way you're talking about from a development perspective. That seems like there'd be a lot of friction just doing development that way. I overthinking it?
Darin
00:27:07.543
Well, Okay, so if I've got… Let's say I'm working on four different work trees right now, four different features, and I'm connecting into the VMs. What, how am I interacting with them the best way? What is the state of that right now?
David
00:27:20.633
For our use case, we use VS Code and it has the ability to use remote connections. So it basically just SSHs in and then installs a dev server, and it feels like you're running locally. You can't really see any difference. If you've got four or five workspaces going on in parallel and you're doing really heavy things inside each one, then you're gonna start to notice it. It's gonna come up against the laptop resources. But if you have a well-specced system or maybe you even run VMs in the cloud and just remote into them then it feels like working locally. It's all managed through VS Code, and other editors can do it as well. Zed does this and I'm sure the other JetBrains IDs do this as well.
Darin
00:28:03.303
So it feels like we're just going back to working in terminals again, except with a little bit better UI
David
00:28:08.903
Yep, using VS Code as the GUI or yeah, or you could just SSH in directly and then you can install Vim or use Cloud Code directly or whatever CLI you, you wanna use.
Viktor
00:28:20.066
I suspect that where we will be going is the-- before AI it was mostly, at least from my experience, as you said, you connect to it one way or another, doesn't matter. It feels local, you just execute it somewhere else. But that feels local, I think that, that will be disappearing because it makes perfect sense that if you already have X dev environments, why not have Claude code in each of them and you just Basically you get rid of local ID or whatever you're using in SSH and just do it
David
00:28:51.296
Yeah, absolutely. And I think we're seeing this isn't my workflow yet, but Code's pushing people there. There are other editors like Warp used to be a terminal, a new terminal, and it still is, but now it's more of an agentic development environment. And VS Code is pushing people to their new agent view, which gets rid of the editor. There's no real text editor as the default. It's just a list of your chat sessions, and then you can see a diff of the changes. And to me, that's still a bit uncomfortable. Like, I'll use that every so often, but that's where things are being pushed, and that means that the editor itself can multiplex all your SSH connections across wherever the environments are running, and you can just see chat sessions. And behind the scenes each chat is SSH'd into that environment and is doing stuff with Claude or Codex or whatever. But from the user perspective, it's just a list of chats
Darin
00:29:43.684
Isn't this where typically a platform team would come into play, right? If you're thinking about the remote compute. But now if I'm on a desktop, usually the desktop minus basic MDM is completely controlled by the developer themselves, more or less. Are we basically shifting left now platform all the way down to the laptop?
David
00:30:07.043
I think if you're running things locally, then the developer has more control there. And for smaller teams, you just don't have a platform team to help you with that. But I-- as you get to larger companies, they're gonna take over a lot of this. And I think that's already the case at enterprise organizations that already have cloud environments, is they don't want any of their proprietary code, any data, anything on the actual endpoint itself. And the endpoint really just exists as a way to remote into the real development environment, which is running somewhere in an approved cloud location. And really then it comes down to latency, because you really don't want to like, see your keystrokes and have to wait as it goes through to, through the cloud. But that's, that's the platform team. That's their responsibility. It's providing that environment, making sure that it's got everything installed. One of the nice features about Codespaces was that it would-- they would do things like pre-builds, where you would commit and something would move into main, and then they would rebuild the base image so that when you needed to spin up a new one, everything was already pre-installed and it could start up in a couple of seconds. 'Cause otherwise, if you're having to do your dependency installs and all those kind of things, then it can take a long time to create a new environment.
Darin
00:31:22.634
This takes me back to the point to where I don't need hardware on my desk anymore. Just give me a Chromebook, right? Or a green screen. That's all I need.
David
00:31:35.208
That's right. Yeah. And like arguably the Chromebook was too early for that, and they didn't have the, didn't have the UI for it
Viktor
00:31:44.764
Feel that the problem in the past with adoption of those tools is that, hey, m- my machine is good enough, more than good enough to do whatever needs to be done. And then I'm faced only with limitations. But now that I feel that we are going more and more towards, oh, I'm not working on this feature, I'm working on 57 of those features, and the tests are being executed and it's being run, and this and that is happening. And I think that now we are getting to the point where actually nobody's laptop. At s- at some point not, not so distant future, nobody's laptop will be good enough, It will be almost like a physical limitation. Now it's not anymore the question of pref- or of your preferences and company's limitations and what so not, but simply it cannot be done.
David
00:32:33.363
It was Apple Silicon that really made me question why am I paying all this money for cloud environments? When the M1 chip came out this was, I don't know, four or five years ago now. That was also just around the time that there were a load of interesting startups around this cloud IDE concept, and it's when Codespaces first started as well. And whilst there were these real benefits of being able to very quickly create a new environment that's t-targeted to a particular branch to be able to create and destroy them quickly, they cost so much, and the provisioning of them was so annoying to, to get up and running. You spend $1,000 on a laptop and and the M1 chip, and it was so amazing. It could do so much stuff with amazing battery life, amazing performance. It just didn't make sense. that was the world where you were just running one editor and doing one thing and focused on, on one task, that's no longer the case. I think 57 might, you might, that might be so stressful, you trying to keep track of everything. But I think the point is is valid, is you're gonna overload your laptop memory first probably
Viktor
00:33:39.340
there is also not, apart from overloading, there is a problem even if you work on two features, let's say in parallel, same project. Okay, ports are going to conflict, when it runs, there's gonna be all sort, those sorts of conflicts the moment you start working on more than one
David
00:33:58.300
Yeah, I think some of those things are solved by the editor. Like the port conflicts can be solved. But I think for me, it's just the context shifting because the agent can go away and do stuff for a while, and then it's gonna come back, and you've got the option to just ignore and just not read the code and open a pull request and be done with it. But I think most engineers are applying their skill at the beginning and telling the agent what needs to be done and maybe guiding it a little bit, and then at the end and reviewing and potentially guiding it to do slightly thing, things slightly differently. But then you still got a context switch. You've gotta go between the different work streams and understand what's happening. for me, often I'll see it, I'll just pay attention to what the agent is doing, like in the background, and then I'll see it going down the wrong way or trying something when I know what the solution is, and I'll try and steer it in the correct direction. And maybe you can do that for two work streams. You could have them tiled next to each other. Like I say, you get to four, that's a little bit more difficult to see the tiling, but any more than that, I think it's gonna be very difficult.
Darin
00:35:01.460
What if somebody comes to you and says, "Hey, look, this whole npm registry thing, all these problems are just overblown. The packages that were affected were just a handful out of the three million packages." Different question, how many of those are actually active? I go back to left-pad to, which to me is one of the very first in the npm ecosystem. Even I heard of that. By the way, why did that need to be a package? That's a different conversation. But I m- what do you tell somebody? It's like, "Look, yeah, it may have only been a few, but was it a big deal? Was it not a big deal?"
David
00:35:35.850
So for most of them, probably not a big deal, but every so often you're going to get a package that is used by a lot of people, either directly or indirectly, and that was the case with the Axios compromise a few months ago. Because that was such a popular library, direct and indirect consumption of it meant that the effects cascaded across the industry. Only just recently, OpenAI had to revoke their sign-in keys for ChatGPT and Codex because they were affected by the Axios compromise. Not that there was any actual compromise of their environment, but just because the potential was there for it to have happened. arguably, that didn't it didn't actually affect anyone, and it's a minor inconvenience to have to make sure you have updated your version of the OpenAI tooling. But that is a serious project that is used by a large number of people. that is an example of where one package compromise is going to affect not just the developers and who happen to install the package in the couple of hours that it was available but because of the issue with no pinning and because of the dependency tree that just expands massively across all of the packages, it has the potential to be pretty big
Darin
00:36:52.078
And we were talking earlier about, the fix potentially is trusted publishing, 2FA. All those things are great, but yet two of the biggest hacks have happened since then, right? And I call it hack, that just the broad
David
00:37:07.000
Yeah. And this goes back to your defense in depth point is there's no one thing that's gonna solve this. You're gonna need package maintainers having two-factor authentication, and probably not just the TOTP code that gets generated every 30 seconds, but like actually phishing resistant pass keys or physical security keys. Then you wanna have the provenance and the OIDC building. You want to have the multi-stage release process. Then you should change your defaults as a package maintainer or an installer so that you pin the packages and make sure you're using npm ci in most cases. Set up your dev environment to be isolated with dev containers or VMs. And then the real hammer is to set a cool down period so that you don't install anything that was released in the last seven days. And then if you want to start using tooling, there are companies that have firewalls. Like I know we use Socket, which does that, does our dependency scanning, and they have a firewall. There's also one from Step Security, which isolates your GitHub actions and acts as a firewall to stop outbound connections. These are the defenses you've gotta put in place to be able to mitigate all these different holes where people can get in.
Darin
00:38:21.293
You realize everything you just stacked out there means it's supposed to be a real application. I don't have time for that
David
00:38:26.693
Yeah. Or if you don't have, if you don't have time, then just set up a VM and you're isolated, and that really solves most problems because then if you do happen to install at a compromised package, it's contained within the VM and you can just delete it.
Darin
00:38:39.603
I want to go back to the cool-down period for just a minute. You said seven days, just, you just threw a number out. W- what do you think is, personally, what's your favorite number at this point?
David
00:38:49.103
At Archjet we use seven days and we enforce that in our automated Dependabot and all the spam it creates. So you've got a new package update, and then it opens a PR, and then that's also another place where you can be compromised because it's opened a pull request that then runs your CI in your repository with access to environment variables. So it starts there. It starts in the Dependabot config or Renovate config or whichever dependency management tool you're using. We use seven days because if you look back at the majority of these compromises is they last just a couple of hours most cases, and that catches most problems. But there are cases where you've got multiple historical versions being compromised, and that's where the immutable releases configuration option in GitHub is important because it prevents people from changing the r- the historical releases. But if you don't have that set up, then you don't want to be installing older versions of packages which have just been replaced with a compromised version. arguably you could say "Well, they could go back forever. If they've got access to the repository, they'll just edit all the releases," and I suppose that is a risk that you would have to mitigate somewhere else, like in installing in the VM. But seven days is a good middle ground because it is, it would have caught most of the recent vulnerabilities, but it also means that you're not behind on updates, which may well be important if it's a security update.
Darin
00:40:18.768
tell me the mechanics behind that, because I've got a use case for this right now. So let's say I wanted to set up seven days. Does the PR open and basically it just stops? PR doesn't open at all? What's the actual mechanics of making sure that's in place?
David
00:40:34.659
Yeah, it depends on the tool that you're using. So you can set this locally in your-- if you're using npm. It's also supported in other package managers as well, like pnpm has this. But you can set it locally. So when you do npm install package if it was released in, within the window that you configure, it'll just say the package doesn't exist. It just won't see it, and it will stop you from installing it. When it comes to something like Dependabot, we don't use Dependabot but we use one called Renovate. And Renovate is if you've ever been to a GitHub repository, and they've got a, an issue that has a dependency dashboard, and it lists all of the dependencies, that is what Renovate gives you. And so it splits up all the dependencies, and you have little tick boxes, so you can choose to open the pull request, which I think is a good first step. Not opening them automatically but doing it deliberately. And when you configure the cooldown period, it will split the dependencies up. It'll have a little group at the bottom that lists all the ones that have an update but don't yet pass that cooldown period because you can optionally choose to then open them and bypass that cooldown, which is important if there's a security update that you need to apply immediately, and you need to make an exception. But then you can see it. It will move up to the list of open dependency updates after the cooldown period expires.
Darin
00:41:53.708
You're talking my language because Renovate is what I use, and I did not know that was even an option because Renovate is one of those things that it's a deep, dark hole once you get into the configurations
David
00:42:05.694
Oh yeah, it's really difficult to configure because there's no way to test it, right? You have to configure it in the, the config. Yeah, and then you have to push it to your repository and see what changes it makes. So it's, it's not ergonomic, but once you have your config set up, it does work really nicely.
Darin
00:42:22.034
There's another thing going on the list today. Great. Thanks for creating more, more work for me today.
Darin
00:42:28.345
You look at the dependency tree inside of node modules, or at least I do, you may not, but I look at it and go, "Where is all this stuff coming from?" Because again, coming from the Java s- ecosystem, I see a .m2 directory, and there's all the thousands of .jar files. It's like, okay, great. Now I'm, I'm doing a fresh Maven install. Perfect. I'm gonna be downloading the internet. I'll see you tomorrow. The amount of transitive dependencies that exist in the JavaScript ecosystem, at least in my opinion, so far exceed what's in the Java ecosystem. Why is that? People just keep on building on top of just the little things that they find. It's like, "Oh, I just need to make this one tweak," or, "I just need to make this other tweak." is that cultural within the ecosystem?
David
00:43:14.106
It seems to be a particular problem in JavaScript, and maybe you can blame the standard library just not being great for a very long time. Nowadays, there's a lot more in there. Also the type of developer in JavaScript has typically been less experienced to begin with. because JavaScript just allows you to do anything wherever you like, however you like, with no rules, basically, it means that anyone can work on it, and so there's less discipline. But that's-- those are kind of all ad hominem attacks at the-- on the individual developer, and that's not in… Like, there's some truth to it, but it's not en-entirely fair. But the… I suppose over time, more and more people have been creating packages. In the past, it was you just didn't wanna reinvent the wheel. The obvious one not to use is like, the isOdd and isEven calculation libraries. Those are the worst because it's literally just a couple of lines of code. But if you have Axios, for instance, that is a good example of something that probably should be a library because it deals with fairly complicated e-edge cases and all sorts of things to do with making web requests, and you don't necessarily wanna write that from scratch. But I feel like this is something that AI is-- has the potential to change for the better because it means that rather than bringing in a full library with every single edge case ever handled, you can direct the AI to write something for your very specific use case, and you can realistically have one hundred percent test coverage of it, which in the past would be, a-aside from maybe it's fun or it's a side project it's difficult to justify spending the time on re-implementing a lot of things from scratch. Now, there's gonna be stuff you never wanna do that with. Like, Don't implement cryptography. Don't get the AI to implement cryptography. Just use a library. But there are many things now that it's just better to have your AI write it for you, or you could write it yourself. Google, for a very long time, has always vended all of their dependencies, and that's something that almost nobody does. it means that it gives their engineers more of a burden to install something from the beginning 'cause they know we bring this in, and we're gonna have to maintain it and vet every release and look at all the code," and no one really does that. Google has an unusual approach to it. But maybe that's something that developers should think about is well, if I'm gonna have to maintain this as a dependency, I might as well just bring it in as a package we write ourselves and take responsibility for the things that are in the code.
Viktor
00:45:43.383
I feel that AI will be demise of quite a lot of open source in terms that, at least in my case I don't feel the need anymore to Google and find which package I need unless it's considerable import. I'm not going to rewrite the, I don't know, like presentation framework or whatever, but simply I don't bother with those things anymore, right? You're gonna write it in a few minutes. Why would I… it takes less time for you to do it than for me to find it
David
00:46:15.804
Yeah, absolutely. our SDKs at Archjet, we try and have zero dependencies because when a customer installs our SDK, we want them to only have to trust us. And whilst we are a library, we add value because we do things like maintain-- like we, one of our features is bot detection, and maintaining that is an arms race. It's never done, and it's never 100% there. We're always doing new things and maintaining our dataset and changing our approaches to things. Then that's the kind of thing you want to delegate to someone else to do. But a lot of the things, like when I look at our renovate list of dependencies now, there are some that are major version changes that's gonna break what we do, so I'm gonna have to put time into fixing the breakage to update to the new version. Or what I'm actually doing most cases now is just taking the use case that we have, and we just implement it from scratch, and we remove that dependency from our project entirely. So we have a very specific version rather than spending the time doing the upgrade, we'll just spend the time getting an AI to implement that for us.
Darin
00:47:16.729
AI is going to solve all of our problems is what I'm hearing. But will it solve the dependency problem that we're seeing? I don't think so
Darin
00:47:24.506
I want to stick with Axios for just a second. I've been working on a Towery project desktop app, and it initially brought in Axios, and I looked at what it was doing and went, "Wait, you're just making an HTTP call. Isn't there a standard library for this?" "Oh, yeah. Yeah I'll just use that." Great. Thanks. Get rid of that dependency." It's like there were already too many dependencies. It's like I don't need one of the ones that already got whacked a
David
00:47:45.934
Yep, Yep, 2026, you can use Fetch. Fetch is built into Node, it's built into all the JavaScript runtimes, and it does a great job. But if you were go, go back a couple of years ago, it wasn't standard and that's where something like Axios would come in. But this is an argument for a good standard library. Go has an amazing standard library as does Python, as does Rust. There are plenty of languages that have really good standard libraries, which means you don't have to pull in third-party packages for a lot of things.
Darin
00:48:24.448
So we've talked about Arcjet. Can you explain to us a little bit more about what Arcjet really does? 'Cause obviously I got it completely wrong in the beginning
David
00:48:31.758
Yeah. So Artjet is runtime policy enforcement. So think about um, let's use an AI chat application as an example. It's a web application. You submit a form, and then maybe the AI goes off and does things behind the scenes. Maybe it makes a web request, a fetch request. Maybe it reads a file, and then it gives you the response back, For that kind of application, you're going to need to do things like budget controls to stop users spending all your tokens. You're gonna need to prevent automation from attacking the application. You want to protect against prompt injection not just on the incoming chat from the user, but also indirect prompt injection. Let's say the agent pulls down a website, and the text has prompt injection in it. You need to detect that and enforce those policies consistently wherever it's deployed, and that's what Artjet does. It allows you to do that in code. You bring it in as a dependency. You write your policies in code. You can then configure them through a web UI as well if you're a security team. But the developers will implement in code security as a feature and build it in because you have… This is where you have all the context, so you know who the user is, you know their billing status, you know the quota they've been assigned. You know that whether they're your most important user, and you shouldn't just block their requests. And so we help developers implement th-those kind of security features.
Darin
00:49:48.147
Most policy engines I've seen are using something like CEL or otherwise to implement the rule sets. What are you doing inside Archit?
David
00:49:58.512
So if you use our JavaScript SDK, then it's JavaScript or TypeScript. If you use Python, it's in Python. That means the policy lives alongside the code it's protecting. And what we're trying to do is get developers to think about security as a feature that they're building and integrate it into the f- flow
Darin
00:50:15.317
I don't think that's a developer problem. I think that's the person that's writing the business requirements document problem uh, because if they're not thinking about it, they're not gonna budget the time for it, and a developer's not just gonna drop it in
David
00:50:26.137
That project manager or the product owner is gonna work with the developer to do it, and that's where we think the collaboration should be. Often what you end up having when you don't think about it that way is that the product owner works with the developer and they forget the security team who are also involved, and then it ends up getting deployed without any protections, and security has to force it on or force the engineers to then try and talk to the product owner, and you get this… It becomes a real friction point whilst you have a production application that doesn't have the proper protections. And so we're trying to bring this closer into the application development flow.
Darin
00:51:01.737
But again, the developers won't do it unless they're told to do it. So bec- well, 'cause it would have to exist, right? The po- because the policies, I'm assuming, don't stand alone. Is there some sort of policy engine, server X, Y, Z, whatever
David
00:51:14.583
Yeah, they can stand alone. So y-you can apply global policies through what we call remote rules through our web UI. So all the developer has to do is install ArchGen and then call it and make sure that it's being called as part of your application flow. And they can ignore the policies. They can let the security team that define them remotely. But what we would like people to do is define them in code because then you can comment them, you can write tests against them, and they can be part of your pull request flow. They can be reviewed. They get in the Git history and just be part of the development life cycle
Darin
00:51:46.608
You can check out Arcjet at arcjet.com. That's A-R-C-J-E-T.com. All of David's information will be down in the episode description. David, thanks for being with us today.
David
00:51:56.589
Thanks a lot We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox.com/contact. If you subscribe through Apple Podcast, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox