Ricardo
00:00:00.000
that's the whole idea of, continuous integration is that you do this so often and so repeatedly that when you have integrations problems, they are small, you can fix it ASAP and move on with your work.
Darin
00:01:10.670
Viktor. We have been in the space of CI CD for quite a while. It's no secret that I work with the Jenkins project. It's no secret that you work with the Crossplane project. I'm sort of dumping that into also a CD tool. I know that's a stretch, but, you know, it makes things so, right? It is a deployment tool.
Viktor
00:01:31.572
depends how much you can stretch. You're stretching now for miles, but yeah, okay.
Darin
00:01:36.700
Okay. All right. Fair enough. On today's show, we have a guest on with us. We have Ricardo Castro on. Ricardo, how are you doing?
Darin
00:01:46.360
Very well. Thank you. this is a requested guest. So we have requested guests for some while, and somebody requested to bring Ricardo on to talk about something that he actually did in Portuguese. And of course, as being a good American, I don't know how to understand Portuguese. So I didn't even try to listen to it. But, uh, But Ricardo did a talk about the truth in CICD. can you just sort of set it up and we're going to poke holes in it because I have strong feelings.
Ricardo
00:02:13.797
Sure. Uh, so first of all, thank you guys for having me. Um, so this was an interview that I did a while back and we were talking a little bit about the notion that most organizations have, or a lot of people have, that just because I have something automated, I'm doing CICD. that's not my opinion, right? and it was a little bit of a discussion back and forth about that, that just because you have stuff automated, just because you build stuff in an automated way, or just because you deploy it in some automated fashion, that doesn't necessarily mean that you're doing CICD. And that was basically the gist of the conversation that we were having.
Ricardo
00:02:52.408
so CI is a precursor for CD, right? So CI, it's the idea that you continuously integrate, work with the remaining of the code base, meaning that you probably need, although the literature, if we can put it that way, recommend some, at least once a day. that you integrate code and make sure that code is in a releasable state, at least. and then the CD part is the actual part where You either put it in front of users or you're ready to put it in front of users. It will vary a little bit, depending on who you're speaking with. But if you're, for example, if you're basically building code for many, many days and not integrating code, And I'm going to show you how to do it with your, the remainder of your team. This means that's probably means that you're not doing CI CD. There's nothing wrong with that, but this is probably means that you're not doing CI CD because you're not actually integrating code.
Viktor
00:03:40.263
sorry for poking more on this, but does that make CD something different than CI or subset or superset of each other? Okay. Okay.
Ricardo
00:03:51.382
I do think that CD is a continuation of CI, right? So CD is the part where you actually integrate code and. At some point, the lines start getting a little bit murkier, because if for you to actually make sure that things work, you will probably need to do some sort of deployment, maybe in some sort of, some sort of sandbox environment, where you do some kind of functional testing or some kind of integration testing, if we say that like the ultimate goal of CD is putting If you have code in front of your users, your customers, you can make that separation where, okay, in, let's put it lower level environments, internal environments, testing environments, you're still deploying and testing it out and making sure that it works. And then the ultimate goal would be to, at the end of the day, put it in front of the users and actually having users using it and receiving feedback on that.
Darin
00:04:36.533
Now let's go ahead and pull back the curtain just a little bit. At the end, Ricardo is going to have nothing to sell anybody because he doesn't work for a vendor, right? He actually has a real job, right? That's, that's the key part to this. So he has no dog in this hunt. it's interesting that you said that Just because you have automation, you don't have CI. And that's always a fun one to try to convince people of. But you also can't have CI without having automation, right?
Ricardo
00:05:02.865
Yeah, that's true. I completely agree with you. You can in a way say that you are doing CI with, if you do any, everything manually, but that doesn't scale, right? So even if you're a small startup, even if you're like three people in a basement somewhere, at some point you can't do everything manually. So I do agree, with your sentence that just because you have stuff automated doesn't mean that you can do CI, but if you don't have automation, you probably can't do CI even at a very small scale.
Viktor
00:05:28.192
Playing devil's advocate here. If it's, let's say if a developer, writes code, pa, pa, pa, pa, pa, runs tests locally, whatever tests that person has, pushes to Git, uh, deploys directly, kubectl apply. Is that CD?
Ricardo
00:05:45.880
Where's the integration part? So if you're running, if you're the, uh, the sole developer and you're the only one docking at the, at that code base, it's probably fine. Right.
Viktor
00:05:54.555
but I cannot push without, if there are, if, if I'm behind the head and I'm pushing to the main line, I would have to pull. Before I push anyways. So I'm integrating with everybody else.
Ricardo
00:06:07.410
Yeah. You could say that you could do it, but at You don't have a, you don't actually don't need a lot of scale, but for that doesn't work, but for that don't work, right? So if your tests are a little bit slower than expected, you'll probably already have stuff on the main line that you're already, you're not testing it. You're already pushing it. So you do have some sort of integration, but you're actually not making your whole process streamlined, but you could, in all fairness, say that you're kind of doing CI CD.
Darin
00:06:35.782
Let me ask the question a little bit differently. Why do we care about CICD? Isn't that a solved problem a decade ago? I mean, who cares?
Ricardo
00:06:42.462
that's something that we. Or at least a lot of organizations think that it's solved, right? So I'm using Jenkins or I'm using whatever, insert your preferred automation platform in there. I have my builds automated. I have my deployments automated. So I have CICD, it's solved. Though, in all fairness, what I see talking to colleagues and seeing other organizations, Struggling with is precisely that, right? So some engineer went off for a week, uh, or, or even more, like I'm saying a week, like a standard, but it could be like two weeks, a month or whatever, and then they need to integrate their changes with someone else's changes. And while we are living in a more of a microservices world where. There's more separation of code bases. That problem might seem that it's not so prevalent right now, but then you have like multiple servers that somehow need to integrate with each other. So maybe I did some change with an API and now something doesn't work. Or maybe I did some change here and there, and then something else with the database doesn't work. And that's where the problems start to arise. So although we Could go off and work for a month and then come back. These integrations problem will, will appear. And that's the whole idea of, continuous integration is that you do this so often and so repeatedly that when you have integrations problems, they are small, you can fix it ASAP and move on with your work.
Darin
00:07:56.850
I don't know, I agree with you on one thing you just said. It was buried right in the middle. Uh, you said something about microservices and separation of codebases. Most microservice platforms I've seen recently have been monorepos.
Ricardo
00:08:18.490
Yeah, I agree. What I was saying, this is to a little bit to the point that, um, Most, or a lot of people say that, Oh, because I'm building my microservices, I don't have a lot of code integration problems, right? So I I'm working on my branch. So no one is actually touching, even if it's a folder, no one, or we are like two people just touching this code. So I don't have a lot of those problems. It was more of the point that I was going, but yeah, but I agree with you. A lot of these repos now are more mono repos and they're basically working off different folders. If at all. Yep.
Viktor
00:08:49.403
That's similar to what you said before, with my interpretation is the fact that you're using Jenkins or something like that does not mean that you're doing a CI CD. I think that that applies to microservices. The fact that you have multiple applications does not mean that you, you are having microservices.
Ricardo
00:09:06.413
Yeah, exactly. So a lot of organizations mostly have what is commonly known as a distributed monolith, right? So just because you deploy it separately, doesn't mean that you have an actual microservice architecture. So if you have dependencies between your service, so to release service A, I need to service B to do a change. That's not a microservice architecture.
Darin
00:09:26.520
I guess my analogy that just came to mind is just because I'm using a hammer doesn't mean I'm building a house. Just because I'm using Just because you're using a tool doesn't mean you're doing Something that might be related to that tool. Just because you're using a tool doesn't mean you're using the tool correctly, or as the authors intended.
Ricardo
00:09:46.328
Yeah. Yeah. So tools on a general sense, tools should be used to achieve some kind of goal. But in some areas like CICD has been so, the term itself has been so overused that people start to make a direct correlation between I'm using a tool. So I must be doing this practice. Right. And it's not, not always the case. And CICD, I think it's just one example where a lot of people just go, Oh, I'm using. ToolX, Jenkins, for example. So I must be doing CICD, right? just because I use this tool and it's not necessarily that the case.
Darin
00:10:17.677
Or the other case is, hey, I need to use this tool for my goal of beefing up my CV so I can go get another job.
Ricardo
00:10:24.408
we do have a lot of that going on in the industry, right? So we do have a lot of people that even going to another company or I need to. Build up this project that uses tool X, let's call it like Kubernetes, so that I can have in my CV, like one year experience with Kubernetes. So that I like my CV will be beefed up and I'll probably get more job opportunities because I'm using this tool.
Viktor
00:10:44.943
That's still better than the opposite, right? You probably want to have employees that are. Yes, maybe CVE might be the primary reason, but that they're learning new stuff and experimenting rather than, uh, I gave up on any life outside of this company, kind of. I gave up on trying to figure out anything. I'm fine.
Ricardo
00:11:08.052
Yeah, that's true. I'm not saying that, that, that it's necessarily bad. but if that's the only reason, I think that, uh, you probably need more reasons on top of that.
Darin
00:11:17.500
So since we're now in 2025, well into 2025, is there anything new on the horizon? Okay, we were sort of making fun of it, but really CI is easily a solved problem. We've been saying Jenkins all along. We'll say GitHub Actions. We'll say GitLab CI. We'll say all of them, right? We're not trying to just set one above the other. That is a solved problem. CD, I don't think will ever be a solved problem. There will be tools that will help you get there. But everybody's deployment environments are different. We can say CI is a solved problem because we're building probably, and again this is going to show my bias, we're building a solution. Either a container image, we're building a WAR file, we're building a ZIP file, we're building some sort of ZIP that's going to get stuffed into, this is one thing we haven't talked about. You can't go straight, you can, you shouldn't go straight from CI to CD. There should be this intermediate binary blob repository of something before going and doing the deployment. Agree or disagree?
Ricardo
00:12:25.343
I agree, I think you should build some sort of artifact, whatever it may be, a WAR file, a, uh, container, whatever it may be, a zip file, whatever it is, and that those things should be versioned, so, uh, as well, uh, so I agree with you that that's the goal of CD, so that you produce some sort of artifact that then you can somehow deploy it in whatever manner you prefer.
Darin
00:12:47.265
So I guess my argument in that is no matter which tool that you're using, everybody can get to a container image, a zip file or whatever, right? There are tools to get those things there. That is going to be true all over this planet. What's not true is once you get it into that repository, even if it's a flat file system, getting it out from there and getting it onto whatever running system that software needs to be on. There are tools to help with that, but I don't think that will ever be a solved problem.
Ricardo
00:13:18.180
I also don't think so. At least looking at the current landscape that we have, the amount of tools, the amount of Different ways. Also, if you work in a highly regulated environment, you will have constraints that will not allow you to do some things or force you to do things in certain ways. so I think that will be a lot of variability, in how you deploy stuff. So we might see some convergence on, let's call it like five, six, seven, whatever number, uh, is reasonable ways to do it, but I don't think it will be like one silver bullet, to actually, this is the way that we should do it.
Darin
00:13:52.195
You're misappropriating this is the way, unfortunately, from Mandalorian, because I don't think there can be a way. think the only way is making sure you've built an artifact. if you're not producing an artifact, you're not doing any of the above. Right? Because the output of a CI, in theory, is going to be an artifact, and the input to a CD is going to be an artifact.
Ricardo
00:14:16.902
That's true, that's true. I agree. I agree with that. So, and, and I've worked in the past doing deployments where you do like a Git checkout with live running code and then doing a restart of a process. And that, as you guys can imagine, that, uh, lead, led to a lot of problems. So I do agree that the, the output of the CI should be some sort of artifact, whatever it may be. And then the input for the CD would be, should be that artifact, that you would then have multiple ways to deploy that.
Viktor
00:14:41.580
if output of CI is an artifact, that means that deployments to ephemeral environments are happening in CD, right?
Ricardo
00:15:04.919
Yeah. So just, okay. Going back to the, the, the, the, the thought that you were, uh, in the, in the previous point where, uh, I was touching a little bit on maybe you are like, you are using sandbox environments or performance environments, testing environments to test that, that artifact, just because the artifact has been built, that doesn't necessarily mean that it is ready. Um, so it means that probably you produced an artifact that maybe you are going to do some rounds of testing. It could be performance testing. It could be a functional validation. It could be some sort of smoke testing. And then at some point, the CI process will say, okay, this artifact is actually ready to go into production because we, um, Did all of these tests. We tested, we had manual testing, we had, uh, some, QA engineers come here and do, did some smoke testing, some validations, just because the artifact is built, it's not mean that it's necessarily ready for production. You can. Go to production and have some sort of deployment strategy that will allow you to collect feedback and roll back and roll forth. But the, um, to your point here is that that doesn't mean that just because the, I did a Docker build push to some registry, it's ready for production.
Darin
00:16:17.657
Well, you've never seen my code. Mine is always ready for production. I think the problem, though, is we've set it up to where there's just these two things. CI and CD. And bottom line, that's just not true. There's that whole magic in the middle. It's sort of that, think about, you know, this is a sandwich. I don't like using this metaphor, but it'll work. We've got the sandwich. We got a CI slice and we got a CD slice. And the piece of cheese or the piece of meat or the piece of vegetable that's in between is this artifact repository. But the problem is that artifact repository isn't just an artifact repository. As you're laying out, there's this testing phase. So what I've seen some people do is they'll take whatever the artifact is and stuff it in the most bottom layer of the repository. And they stage it through different environments until finally it's in the environment that Production can actually pick it up. It's sort of like you land it in the basement elevator or in the basement and you ride the elevator up until you finally get to the top floor to where then you can go ahead and send it out to production. Should we have yet another acronym for that section? Like that little piece of cheese?
Ricardo
00:17:33.492
I think that at this point, and because CICD will defer A lot of from company to company and depending how you speak to maybe, I'm not entirely sure, about that, but it kind of makes sense that because the reality is that for most companies, that's what happens, right? You do a PR, there's some kind of unit testing run that runs, there's code reviews, you go back and forth a little bit. Somehow code gets merged to your main line. You produce an artifact, be it a Docker container, a WAR file, whatever it may be. And then what usually happens is that organizations will have a. Several layers where that artifact will work. Keep pushing forward, like some sort of development environment, sandbox, testing performance until it arrives in production. where we draw the line between where CI or, uh, where CI ends and CD starts, we could argue that maybe that that's all CI because we are, we are still validating that the artifact is actually shippable, or we could have some intermediate layer where, okay, CI ended because we did some pre validation that the artifact is okay. And then we have some sort of. middle ground here where we are continuously testing.
Ricardo
00:18:43.454
That's true. Yeah. But, uh, if we, if we want, if we don't want to introduce any new, names, we could say that CI only ends, when we're ready to go to production. Even if we keep just bumping that artifact from stage to stage, to sandbox, to, pre prod and far. environment, but it wouldn't hurt if we had something, uh, in the middle.
Viktor
00:19:05.193
ends when the artifact is ready for production. And given that, that means that CI was deploying somewhere else before that, and the process deploying of somewhere else, uh, to somewhere else and production is give or take the same. why would technically anybody have CI without CD? Why would there be a single company that says we're doing only CI?
Ricardo
00:19:30.443
I actually don't know the answer for that question. Apart from you're operating a. I think the most important thing is to have a highly regulated environment where you can't deploy in an automated, like, in an easy way. So maybe you need some sort of regulator to say, yes, you can deploy this. And even then you could argue that we're still doing CI. We're just waiting for the regulator to, to say so. so the point that I'm trying to make here is that if you're going out of your way to actually implement. CI and having everything automated, so CD should be the consequence, right? Because you've already done so much, you're putting so much effort into this to making sure, or as best as you can, to be sure that does what you're expecting it to do. So Let's roll it out to, to our users. Of course, then there's the whole CD part where if you're using Canary deployments, blue, green, and whatever, you have ways to mitigate that. And that's a whole different discussion. But yeah, I would argue that if you're going out of your way to actually do CI properly and doing those many, many times a day, integrating code, putting it out ready to, to ship, you should probably go the extra mile and just deploy it.
Viktor
00:20:33.814
Because then, I guess, I could claim that, according to those definitions we laid out so far, that whenever somebody says, I'm doing CI, but I'm not doing CD, that person is probably not doing CI either, kind of just
Ricardo
00:20:49.474
Yeah, would agree with you. that's probably where people fall into, the category where they're basically saying, I use this tool that has this thing automated, so I'm doing CI, but not really right.
Viktor
00:21:00.221
Yeah, because then it would be, my head would interpret it as, you're doing, you say you're doing CI, but you're afraid what will happen if you deploy to production. That means that outputs of your CI are not reliable.
Ricardo
00:21:15.736
Yeah, you don't trust them. At least you don't trust them, right? Maybe you need some extra tests or you understand that you're not testing like this thing or that thing. you don't have enough trust, both on the integration part, let's put it this way, the testing part, nor in the process that you have to deploy it, so that if something If your system is not okay, you are able to recover quickly. maybe because you were supposed to just a subset of users, or you have an easy way to just say, okay, something's wrong. I just roll it back to the, to the previous version. So, yeah. Um, so I agree with that point there.
Darin
00:21:48.033
Two words you're using over the past couple of minutes, trust and easy. Aren't we paid to make the hard things easy? And aren't we paid to make things trustable?
Ricardo
00:21:58.456
That should be the goal right? we are paid to actually write software, that our users can trust. and to deploy them, but that's not always necessarily the case. So, and in a lot, in a lot of companies, there are lots of stakeholders there are lots of interested parts, depending on the company that you are working for, you have more pressure from security, from product, from whatever, insert the vertical that actually, has a lot of sway in your organization. And it's not uncommon for certain engineering practices to, Be relaxed a little bit, so that you are able to achieve some sort goal that not doesn't encompass necessarily, the whole slew of things that you would like to have covered.
Darin
00:22:41.520
Let's set up a scenario. Let's see how you would solve it. Because you said the word security, so I'm going to throw security under the bus because they're always fun to throw under the bus. And then they make my life miserable after that. So let's say you've been working on an app and you've got a dependency and that dependency has a security, a high CVE, like 9. 5 and higher, right? So that's been deemed within the company that that has to be fixed. The library author has not yet released something to fix that CVE. We've got a major bug in our application and we need to get a new version of the application out, but we can't put the version of the application out because it still has that library that's got the CVE and we don't allow the, we get into this circular problem. How would you solve that?
Ricardo
00:23:30.252
at some point there needs to be a discussion between security and whatever pro whatever product means within your organization. Right? So that there will need to be a compromise between both parties. we either don't update and we Don't do that release, or we accept the risk. And there's a lot that we can talk about, about risk management and risk assessment. your case, that will probably be a discussion between engineering and output, like security, just for the sake of the discussion, not being part of engineering, That I think it is, but a different part and product development where we actually make an assessment. So how serious is this EV? Can it be impacted? is it accessible from, from the exterior? how can the users exploit it and what would be the drawbacks and the downsides of not releasing this, new feature, can we wait until, the fix has been, uh, has been put forward or can we actually put some company resources behind that? If it's an open source project, can we actually go there and help the maintainer to actually fix it?
Darin
00:24:27.251
That last part you said there I think is the key, is can we as a company help the maintainer? Let's just go down this path. If you're, as a company, are using open source and not giving back whatever that means, money or time. Uh, you are not stealing. I won't go as far as to say that, but you're not being a good community member. You're basically the person driving through at the drive thru and it's like, uh, Hey, thanks for my food and driving off. Notice you didn't pay for it. That's not so cool, unless you're set up to be, unless you're driving through like maybe a food line, right? You're driving through a food line, uh, except you've got the resources to help actually pay for that food instead. How far have I stepped off into it now?
Ricardo
00:25:19.653
No, I agree. I agree with that. And there has been at least in the last couple of years, a lot of discussion about how to make money from open source and a lot of companies Coming into GitHub, for example, and demanding from the creators of some sort of framework or library, like I need this fixed right now, or I need this feature implemented without contributing back, either with code or some money contributions. So I do think that, if organizations have resources and have the way to actually contribute bug fixes or even contribute with features, I think it's something that those companies should do. For example, the company that I work for, we can, we regularly do that if we. Find some kind of, bug, we actually contribute back to the community. So I do, I do think that's a good path to go with.
Darin
00:26:01.666
is that your first, without talking about who you work for, is that your first reaction as a company to, we actually know how to fix this. So we'll put in the PR. And you're willing to go through any of the other pain that you might have to signing a CLA or doing whatever. You're willing to do that as a company.
Ricardo
00:26:19.866
Yeah, as a company, we're, we're, uh, for the, at least for the core things that we, that we use, So if we can wait it out, I think that the company, if it's not like something very big, the company will not allocate those resources. If it's something that we actually deem important, important, like it's something that preventing some sort of release, or it's something that it's every really high security risk, the company will have no problems whatsoever into putting those resources and say, okay, we'll go there and we'll do that bug fix, or we will implement that thing that is actually. giving us a lot of, trouble and we give it back to the community.
Darin
00:26:49.684
That just feels like the right thing to do. Why is it so hard for people to sort of understand that?
Viktor
00:26:56.699
I feel that that's actually a wrong way to look at it, right? We very often in open source talk about the right thing to do. And I think that companies should should be selfish. I think that they should be working with open source for purely selfish reasons. And realize eventually that those selfish reasons lead them to, to contributions, right? You contribute not because it's the right thing to do, but because you need that CV fixed. Because you need that feature, and it's easier, it's cheaper to put it into the project than it's Then abandoning it altogether and starting something fresh because of a single feature that you're missing, and so on and so forth, right? I feel that when we start thinking about open source from very selfish perspectives, actually, we will move forward faster, to be honest. It's just that companies do not understand how much those contributions, doing contributions, would actually benefit them.
Ricardo
00:28:01.613
I think that the companies, or at least the vast majority of companies that do it, do it in that way, right? they are not willing to, so they have even have like a bug fix that is something so serious and no one's actually tackling it and they just go and do it, or it would be awesome if that framework that they're using had that feature and I don't want to rewrite my whole code base to adopt another framework or another library just because that feature misses, so I'll go there and do it, At least most companies, they just go down that path. If they really have to, if they can't wait it out, if it's something that they really need to do, and maybe that's the solution. Maybe it's like if companies internalize the fact that if I go there and actually fix that thing. the giving back to the community is a by product that we all benefit from, but it's, maybe it's easier to convince senior management, say, okay, if we go there and we allocate a few engineers for a sprint or something like that, to actually implement that feature, we will, as a company will benefit a lot because it will give us that thing, or we will be able to deploy that new feature that doesn't have that CV. So maybe that's a solution.
Darin
00:29:03.848
think Viktor, you made my point for me. I was saying that, you know, the right thing, well, right thing is all in perspective, speaking as an open source author, the right thing is, okay, I want people to help me out. But I think Viktor's is even more important that if your senior management doesn't understand the value of open source within the company, then engineering hasn't done a good job of. Really saying, look, if we had to get rid of all of these libraries, if we had to get rid of this software that we're not getting vendor support on, we would, and we had to do everything internally, we would not have a company and then CICD wouldn't matter anyway.
Ricardo
00:29:49.512
Yeah, that's true. And we, and we had, a few examples recently where some companies that were building open source software and providing some professional service, services around that software, actually taking those projects, in some way in a private fashion, or at least not contributed from the, from the community. And I don't know if senior management has realized that, In some cases, if that project didn't exist, or if it was completely, taken private and the amount of money that they would have to pay for that, it would be a lot more expensive than just contributing back to the community.
Darin
00:30:26.759
guess the only way you can get senior management to understand anything is put it in terms of dollars and cents.
Ricardo
00:30:32.197
Yeah, that's true, that's true. And I find out that depending, of course, this will vary immensely from company to company, but when you put some sort of dollar value behind something, even if it's some, something high level or highly skewed into something, it makes some conversations easier because at least people that are Not so accustomed with engineering. We'll say, okay, so doing this will cost X amount of money or not doing something will cost X amount of money. Okay. We can work with that. then just trying to understand some why behind it. So people go straight to the dollar value.
Darin
00:31:06.428
So let's try to circle back to what we were originally talking about. I sent this down a bad trail here. The truth of CICD. People always want to know what tools, what tools, what tools. I'm a Java person, so the tools are Maven or Gradle. If you're a Go developer, the tool is Viktor. Help me.
Viktor
00:31:31.628
would makefile always. I mean, Go itself, actually, and then wrapped in a makefile, yes.
Darin
00:31:39.338
In a make file. Right. Do you really need a lot of tools on the CI side of things? Let's leave CD out of it for a minute. Do we, do we really need a lot of tools? Isn't, isn't make good enough? I don't like make, but isn't it make good enough?
Ricardo
00:31:55.479
I don't think it's, you don't, you'll do need a lot of tools. It's, it's more about the approach that you're, take, um, on how you build your software, like you write and you build software than necessarily if you're using Gradle or if you're using Make or if you're using a shell script or whatever tool that you prefer. so it's about that. If you don't have a way to actually integrate your changes with the remainder of your team, with the rest of the code base, can you really say that you're doing CI CD, right? If I'm working for a month on my own and everyone else is doing the same, Will we not have problems when we have to integrate code because we're touching the same code base? Then do we want to spend two more weeks just figuring out how all the pieces fit together? Or should we be doing this in small increments in this idea of doing small changes, validating it, putting it out there and continuously doing that?
Ricardo
00:32:55.783
But you could, in theory, you could, right? Uh, you could, do it manually, like, like Viktor was saying in the beginning, right? So I'm doing this test. I'll pull, I pull from the main branch. I run my local tests. I do kubectl, apply, done, right? I could do it manually, but at some point it doesn't scale.
Darin
00:33:10.471
So if we're doing something that doesn't scale and we're unwilling to change, Is that a problem?
Ricardo
00:33:20.321
yeah, yeah, yeah, exactly. Could be. Maybe you don't have the, like Viktor was saying, maybe you don't have the resources to do it. So you're, you're just stuck. You don't have the resources to actually allocate it to some engineer to automate some, some kind of stuff, or you don't have the resources to build some kind of infrastructure to test things out or whatever it is.
Viktor
00:33:37.444
I meant the other way around when I said, depends how much money you have. If you have A lot of money to burn. You can just solve any problem with putting more people. Hey, this can be done by 10, but I have 100 people doing it. I don't need to invest in anything else.
Ricardo
00:33:56.195
Yeah. And you will also see that a lot in companies that are probably trying to get some market share, right? So they will deprioritize, for example, some sort of automation or some sort of Call it CICD pipeline, just so that they can focus their engineers into building the next feature, the next thing, because they're, I'm using this just as an example, we can probably think of a lot more, where you will just say, okay, I'll allocate, I have a lot of money, but I need to allocate these resources into this thing that I think it's a lot more priority because I don't know, my competitors are too fierce and I really need to stay ahead of the curve. but again, like you said, we could throw people at the problem. People at the end of the day will probably cost us more, than starting to, at some point, automating some of these things, I think.
Darin
00:34:42.814
I'm going to add in a, maybe a different analogy here. I saw a meme recently that showed the number of space launches that are occurring by company and, or, you know, delivery of payloads, whatever the exact phrasing was I don't remember. And not surprisingly, SpaceX was at the very top of that by a wide margin. And Boeing was at the very bottom. So I think about the two different ways that those two companies exterior for me, just looking out at, you know, as a space nerd, looking at it, SpaceX launches and reuses, Boeing on the other hand, launches and trashes. to me, that's the manual versus automated part. I mean, it's, it's a very. Poor analogy, but if I'm able to reuse and just keep, because I'm exercising that over and over and over again versus the manual part, it's a one shot. I mean, that's what we do a lot as developers too, is we do all these little one offs. Then unfortunately, all the one offs get wired together to turn into the automation. Is that a bad thing?
Ricardo
00:36:03.948
Yeah, I think that's the, the path that probably most companies, go with. It's like, you start doing some stuff manually, you start to automate, some parts. Maybe there's some thing here that I wrap around a, a shell script. Now I have a bunch of shell scripts. I start making some sort of a pipeline because each script does a small thing. And at some point. you probably have like a whole pipeline that builds, tests, and deploys. The problem, like we were saying, like this, this part here of doing it manually or automating is that when you don't do this, this iterative process and you keep doing some, some stuff manually, like you said, you trash it. You don't reuse a lot of this automation and humans are faulty by nature, right? So there'll probably do some mistakes along the way of something that should be done. the same way every single time. So they will probably do a mistake. They will crash and burn something that will probably cost the company a lot more money than if they invest in that automation that will do that thing over and over again in the same way.
Darin
00:37:07.075
But if we still can't get to automation, what hope is there for us? Brushing up this new CV and getting a new job.
Ricardo
00:37:14.713
Yeah, I think it will depend on the company, right? we have companies from different sizes, from different shapes, but it comes, all comes down about culture, and the way that you want your engineering to evolve. That doesn't mean that they are better or worse. It's just the way that they are set up and the way that I, they want to work. in a certain way. so engineering has to do, it's part of actually showing the, benefits of doing things in a certain way. This is if we deploy more often in smaller batches, we probably can reduce when we have problems, they are of a smaller scale, we can introduce features at the end of the day faster because we will keep introducing things into the system. there will be organizations that don't want to do that. So maybe, uh, at some point in your career that then when you are in a company, maybe brushing off your CV and going somewhere else could be an option there.
Ricardo
00:38:12.296
no. Yeah, I don't think there's a nothing. I don't think there's anything wrong. There's a lot more potential for things to go wrong, right? That, that, that's the only thing, because you're develop, you're deploying a big ball of stuff though. So there's so much stuff in there that there's higher risk, I would say, because you're deploying so much stuff at the same time, and also if you have to fix something, now you have a bunch of stuff that is interconnected, like, okay, this doesn't work, what exactly in this big ball of stuff? Doesn't work. That makes this unexpected outcome to appear. the contrast is, okay, I'm deploying a lot of times per day, for example, or every week I'm deploying small stuff. So if something breaks, I know exactly what was there, or at least I can quickly find what the, what the thing was that, that is live. I can understand it because it's a lot smaller and I can say, okay, so probably the problem is here. I can fix it and throw it back to production again.
Darin
00:39:08.022
I feel like it can be, right? Because if it's, let's say my deployment cycle takes, let's say even five minutes. And that's probably very fast in a lot of environments. Let's say it's five minutes, but I'm getting a flood of, Changes coming in and let's say I'm doing each change as a release, right? Just, just think basic website type stuff for a moment. And for whatever reason, it takes five minutes, but now I've got a hundred things in the queue because I've all of a sudden we had a commit storm and now I'm backed up. Well, the problem with that is I don't, even though I know which order it arrived in, the apply of those may not be in the order that they arrived.
Darin
00:39:54.652
Right. And I'm still waiting for things to get deployed. I mean, it seems like there should be a reasonable, it seems like my batch size should not be smaller than however long it takes to do the, some correlation between the size of the batch and how long it takes to actually deploy it.
Viktor
00:40:21.612
Because let's, let's say that I have half an hour, my process takes half an hour, right? And in half an hour, I made, I don't know, five different pushes, right? The process is finished. What is next in the queue? The last one or the next one? If you understand what I mean.
Darin
00:40:39.442
Yeah, I understand what you mean, but I don't know which one the next one. I mean, that's what I'm saying is if, if we've decided every commit has to go out as a, as its own release,
Darin
00:40:49.542
just, I'm not saying that it's a good thing. I'm just saying that we, that's the decision that we made and that we're going to stick with that. Then we're going to have problems based on what you're saying. It was like, well, we should be aggregating or picking up whatever the last release cycle. So what you're saying, Viktor, is on the top of the hour and the bottom of the hour, we know it takes 15 minutes. I mean, I'm going to use different numbers than you. We know it takes 15 minutes for everything to deploy. So at the top of the hour and the bottom of the hour, somewhere within that, we're going to be doing deploys. So a scheduled deploy, not just an event based deploy when something hits. Does that work?
Ricardo
00:41:27.036
I think we've done that in the past with the nightly builds, right? So we would like build software overnight and then either could deploy it like the next day or something like that. And I, I think that that could work, but I, but I understand, but I understand what you're saying. Basically saying that if we have like 15 minute process and we deploy, we have like a commit storm just coming, uh, in like, what should we deploy next? what I would say most companies doing would be. Like the next thing. So the things will like to just pile up at some point. If you're getting too many commits, then you can actually deploy. You probably need to start doing those batches, right? So I'll probably do at the top of the hour or the bottom of the hour, just booting them and deploying them as a whole.
Viktor
00:42:10.417
mean, looking at deployment itself, only that part, right? And let's say that you're doing some kind of GitOps thingy, right? You have it configured to look at the head of the main, main line. And it doesn't matter whether you have five pushes or zero pushes in between iterations. It's simply, that's what it is, the last, the main line, the head.
Viktor
00:42:34.488
The problem I feel is it's more in a line of resource utilization, right? Okay, so we might end up having Thousand processes, you know, doing something, building stuff in parallel for no good reason.
Ricardo
00:42:51.093
Yeah, that's true. That will probably, if you have a lot of stuff and you're just building every single thing and deploying it to a, probably, like you said, you'll, you'll have like a lot of infrastructure just set up just to, to do that, those things in parallel.
Viktor
00:43:01.906
but you know, I think that those are such beautiful problems to have. Imagine when you're in a company and say, our problem is that actually an iteration of something too fast. Nobody ever complained about that. Nobody.
Ricardo
00:43:19.029
Yeah. the only asterisk that I would put here, would be like, If you're deploying stuff that is too, too small, you're probably put a lot of defensive code because maybe some feature flags along the way and a lot of stuff that because, okay, I'm deploying this because it's, I don't know, it's 6 PM and I need to go home, but I want to do my CI of the day. so I'll, push this. So I have some defensive code here because this will go to production, but I don't want this activated. So my, what I'm trying to say here is that maybe if we're putting like one or two lines of code, then you'll have a lot of defensive code around the edges. So, okay, I don't need this activated actually is there. I guarantee that everything works, but it's not touching my code. So it, that there might be a point here where you're saying, okay, it even make sense just to like do that checkbox of saying, Oh, I actually did my, I'm going to do a whole commit thing that I put so much stuff around it so that I ensure that this thing doesn't blow up. So that would be one of the things that I will say, okay, maybe, uh, that's not the best approach to go with.
Darin
00:44:18.966
So have we come up with the truth of CICD in this conversation? I don't think we have. The truth is that. There is no truth.
Ricardo
00:44:27.176
Yeah, I don't think if the, I don't know if there's a, a single truth. It depends on who you speak with. for example, if you speak, uh, I'm going to give an example of Dave Farley. he has a very, defined idea of what CI and, and CD is. It's that idea that we've been talking a lot along this conversation, uh, all along this idea of doing small batches of integrating code, testing it and rolling it out to production. I would say that on a higher level, uh, that's what CI CD, uh, is supposed to be, but it will be different from organization to organization. It will look at least slightly different. What I will say is that if you're working. In isolation, right? So I'm going off on my feature branch and working on, on it for weeks at a time, and then integrating it back to the main line. I'm probably not doing continuous integration. I'm doing integration because at the end of the day, I need to integrate my code with the rest of the code base. But it's this idea of continuous. What continuously mean? Is it daily? Is it every other day? Is it every hour? It will slightly different, but I will say like, if it's something like two weeks or a monthly cycle, it's probably not continuous.
Viktor
00:45:34.232
think that the word continuous is very, very wrong. I think it's all about not, not, not working for too long before committing, right? I have, I haven't had in the past as well, projects that actually, we don't work on this project often, right? It could be an application that is not changed in any form of way for a month. But then when we do work on it. Hey, pim pam, couple of hours, maybe a day, push, uh, release, right? So it's definitely not continuous, it's just that we don't spe when we do work on it, we go fast, or commit after a relatively short period of time.
Ricardo
00:46:21.346
Yeah, that's true. I think that's a good, that's a good example. So pretty much every company has that some project that you only touch every once in a while, because I don't know, there's not a lot of stuff to do there or the project is stable, but it's when you work, you don't spend like, let's go for a month or two months or three months just building stuff and then we release it all together. It's, it's that, that speed and getting those stuff out the door as fast as you can.
Darin
00:46:45.275
So all of Ricardo's contact information is going to be down in the episode description. Ricardo, thanks for being with us today.