DOP 60: Jenkins X: Why Good Is Better Than Best

Posted on Wednesday, Jun 17, 2020

Show Notes

#60: As a followup from last week’s episode, we talk about the specific problems that Jenkins X solves.


Darin Pope

Darin Pope

Darin Pope is a developer advocate for CloudBees.

Viktor Farcic

Viktor Farcic

Viktor Farcic is a member of the Google Developer Experts and Docker Captains groups, and published author.

His big passions are DevOps, Containers, Kubernetes, Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).

He often speaks at community gatherings and conferences (latest can be found here).

He has published The DevOps Toolkit Series, DevOps Paradox and Test-Driven Java Development.

His random thoughts and tutorials can be found in his blog

Rate, Review, & Subscribe on Apple Podcasts

If you like our podcast, please consider rating and reviewing our show! Click here, scroll to the bottom, tap to rate with five stars, and select “Write a Review.” Then be sure to let us know what you liked most about the episode!

Also, if you haven’t done so already, subscribe to the podcast. We're adding a bunch of bonus episodes to the feed and, if you’re not subscribed, there’s a good chance you’ll miss out. Subscribe now!

Signup to receive an email when new content is released


Darin Pope 0:00
This is episode number 60 of DevOps Paradox with Darin Pope and Viktor Farcic. I am Darin

Viktor Farcic 0:06
and I am Viktor.

Darin Pope 0:08
And today we're going to build on what we were talking about last week. Now if you haven't listened to last week's episode yet, pause, go listen to last week's episode and then come back. We're not going to wait for 30 minutes while you do that. But in case you don't want to go listen right now, we'll give you a quick 30 second summary. If you are building greenfield business applications today, and you're wanting to deploy to cloud, if you are not deploying to Kubernetes on the cloud, you're making a very poor decision for your company, is what you're doing. That that's that's the gist of what we talked about last week. Now you can disagree with us and that's fine. Because are we saying that if you're a technology company and you're building databases and you're not deploying to Kubernetes that's dumb. No, we're not saying that. What we're saying is business applications, right? I'm building microservices ignore what that really means. Or I'm building some sort of API type stuff. That's what we're talking about. Now, what we wanted to build on top of since Kubernetes, is a platform. We have a question asked, or actually Viktor had I don't have it asked me very often, but Viktor hasn't asked of him multiple times per day. And that question is, what problems does Jenkins X really solve?

Viktor Farcic 1:36
All of them

Darin Pope 1:38
Great. Thanks for listening.

Viktor Farcic 1:40
And thank you, goodbye.

Darin Pope 1:42
Okay, so we're joking, but not really. But yes, we are. Right? Because it doesn't solve every problem.

Viktor Farcic 1:48
No, it doesn't solve it and nothing solves every problem. Whomever sells you something that solves every problem that that somebody is either working for Oracle or is insane as well.

Darin Pope 1:59
I probably should beep that one out.

Viktor Farcic 2:01
Yeah. Okay, yeah we like Oracle.

Darin Pope 2:06
Hey, if it wasn't for Oracle, we wouldn't have had Hudson which gave us Jenkins which then gave us Jenkins X.

Viktor Farcic 2:11
Yes. We would have Hudson from Sun. And it would never get renamed into Jenkins.

Darin Pope 2:22
As much as we love to hate Oracle, they have served a great purpose. We can joke about a lot of other things about it. But Oracle has helped lay the foundation for where we are today.

Viktor Farcic 2:32
Yes, that's that is true.

Darin Pope 2:35
But now let's ignore Oracle for the rest of the conversation.

Viktor Farcic 2:37
No more Oracle. Whenever I want an example, I will use IBM from now on.

Darin Pope 2:42
Okay, good. We have to be careful now because the guy that used to run Red Hat is now in charge of IBM. And he's a good guy. I like him.

Viktor Farcic 2:50
Oh, yeah. And I actually wish him well. I really would like to see IBM steering into the direction of Red Hat instead of eating Red Hat.

Darin Pope 3:04
I'm excited about that. But that's not what we're talking about today. We need to talk about that a different day. But so let's get back to the question at hand is what problems does Jenkins X solve?

Viktor Farcic 3:16
So I think on a first look, it's you know, it is a CI/CD tool. That's, that's how it started, I would say and there are many tools that solve problems, how you do, what is the tool you want to use, you want to use for CI/CD? Right. That's, that's not a really big problem. I think what it solves and I'm building now on the previous conversation about Kubernetes is that once we all agree that you are jumping into Kubernetes unless you're already there, then you're faced with the huge issue of Kubernetes being very innovative and innovation today means many different vendors, many different companies, many different open source projects trying to solve different problems within Kubernetes ecosystem, right? How do you do logging? How do you do networking? How do you do monitoring, and so on and so forth. So we are all seeing tremendous level of innovation in basically almost every aspect of software industry or software delivery lifecycle. But the problem with that level of innovation is you need to figure out what you're going to choose what you're going to you're going to use, how you're going to use it and what will be the processes around it. So I think that what we're trying to do, what Jenkins X is trying to do, is to provide opinion about each step in the lifecycle of an application. Like a simple question, and this is a question that I get asked often is okay, we're actually not the question that I get asked but the statement. Okay, so we are building container images with Docker. We hear the almost every day and then I freak out say, Are you insane? You cannot build sorry, not insane, are you uninformed, you cannot build container images with Docker because Docker can run only on on a node it can only run as a daemon that means that you need to mount a socket or make your containers from which you're building images privileged. That means that you will potentially be much less secure because other people can take control of your Docker daemon and through Docker daemon can take control of your cluster, and you're doing things without Kubernetes knowing so, you're, you're impeding Kubernetes to do its work, how to schedule things correctly, and so on and so forth. So I have like, at least 15 minutes sometimes much longer, longer. ramblings about how that is a bad idea and then comes and then people get tired of me and say, Okay, so what should we do? And then we start discussing for a long period of time, should you use Buildah? Should you use Kaniko? Should you use this? Should use that? Right? It can take just a simple thing, and probably one of the seamlessly simple things possible, how to build a container image can turn into being a complicated decision that requires a lot of information, a lot of learning lots of exploring, to figure it out, because there are too many solutions. Now where Jenkins X jumps in I think, is trying to provide opinions, what are good ways to do certain things. We for example, prefer to build the images with Kaniko because of this and this and that. If you want to deploy your application then yet again, there is 50,000 different ways, different ways how you can define your application. But we say that it's, this might be a good way to do it and so on and so forth. So it is a, it gives you an opinionated lifecycle of your application which you can change. And this is I think also a good thing because we are matching to GitOps so everything ends up being a file, there is no hidden magic and you can say, No, I don't agree with this part I'm going to going to change but you get a working solution for the full lifecycle of your application from idea until production as a base in like 15 seconds. And I think that that's what differentiates it compared to other tools and I'm going to say, one of the tools I worked in Jenkins so I don't do negative marketing for others, but Jenkins does much more things, but it's a blank slate. So you need to figure out what you need what you want to do, instead of starting with a good position.

Darin Pope 7:59
It's sort of like, if you're an application developer, and if you're old like me, you remember the initial release of Spring Framework, the 1.x version of Spring Framework. And now you have Spring Boot which can make things much simpler. So that's the basic comparison of Jenkins to Jenkins X as Jenkins X or Jenkins you can go at it whichever way you want Jenkins X, hey, there's different ways of doing these things. One of the so what is what is the name of that opinionation? Is it a build pack? Is that the right? Or what's the what's the correct phrasing for that today?

Viktor Farcic 8:38
So build pack is a collection of collection of starter templates, right that give you the configuration files for Dockerfile and Makefile and Scaffold and all many different things. But it's, it's it's a that's a kind of like base and then we we start talking about if today, you want to provide certain traceability of what you're doing, you're going to store everything in Git. That means that you're adopting GitOps because Git is the source of truth, and so on and so forth. I'm not entering now deep into the subject, we can do that. But then yes, so GitOps is the way to do we're going to provide the mechanism to do that. ChatOps is a nice thing if you want to record everything happening. So it provides opinions how to, let's rephrase, actually that I think that it codifies good practices, while Jenkins would allows you to, to codify good practices.

Darin Pope 9:37
So Jenkins allows you to codify practices. Jenkins X provides you codified practices.

Viktor Farcic 9:45
Yes, exactly.

Darin Pope 9:47
Is that one of the problems that Jenkins X solves is providing codified best practices as defined by and this is the big question, who defined them?

Viktor Farcic 9:56
It's a open source project. It's community.

Darin Pope 10:00
So whoever wrote it defined it, right? Yeah, no, but unless somebody fights against it, it's now considered to be a quote unquote best practice. That's a question.

Viktor Farcic 10:08
I don't know if you noticed, I'm not actually using word best practice. I'm using good practice. Because best practice Yeah,

Darin Pope 10:15
yeah, cuz there is no, there is no true best practice.

Viktor Farcic 10:17
Exactly. But it's kind of codifying things that and we might argue, because nobody's perfect. But the community thinks are good practices. Like for example, there is a huge move towards codifying DORA, which is based on a lot of data that proves how certain teams are better than others depending on what they do, what they don't do, and so on and so forth. So it's just codifiying good practices. And and the beauty I think that so there is a technical aspect of Jenkins X that we are trying to provide as a product better solution than others. But those good practices, it's not really we don't have interest to promote one or the other, it's more it's purely what we like. Not that much what we have interest in. We have of course, every product has certain interest but they're on in other areas kind of oohh are we going to use Tekton or we're going to use something else from technical perspective, but on best practices is just whatever works best or we believe it works well, right. Now, if you're Kelsey Hightower, you might say, Hey, man, look, I invented good practices in Kubernetes. I don't need you to tell me what are good practices. And then the value of Jenkins X is I think is still valuable but not as high. Right? But if you're starting, if you're unsure in in all those things that I mentioned and hundreds of others, if you haven't made all the choices that you will ever want to make in Kubernetes then it's wonderful because gives you just a quick start.

Darin Pope 12:02
So number one, one of the problems that Jenkins X solves is codifying, codifying, however you want to say it, good practices.

Viktor Farcic 12:13

Darin Pope 12:14
What's another thing that Jenkins X solves?

Viktor Farcic 12:17
It's open source, I mean, I was about to say it's open source that doesn't solve. That's not the solution. Excellent. Yes.

Darin Pope 12:23
You've been doing too much marketing. Okay. Come on. What's another thing it really solves. Now, obviously, the first one that we just said is huge. We're sort of saying we're glossing over it, but being able to say, Hey, here's my application. I want to follow this practice. Go. Right. Just it just follows on from there as long as I play within those guardrails. It just works. So that's that's a good thing. But that only gets me so far. So what's what's what are the other problems that Jenkins X can solve for us?

Viktor Farcic 12:57
But I would say actually, that gets you almost everything because if we say that it solves it, it provides full lifecycle of your application, right? From the creation of the first file in your project all the way up until it is running in production, then basically, the only thing that it doesn't solve is you writing code. So basically, the idea is you write code you push to Git and things happen. And those things that happen, are already predefined. You can change them so that other things happen that that's okay. But with sensible predefined actions, right, let's say. So, now it doesn't solve anything beyond full automation of lifecycle of your application. But most of the rest of the things you're supposed to do, and actually, I think most of the things that 99% of people should be doing is write code and push to Git. What else is there to do?

Darin Pope 12:58
Well, if you're an application developer, hopefully you're not actually operating the Kubernetes cluster.

Viktor Farcic 14:03
But that's the thing. If you want to operate the Kubernetes cluster, you're still going to change something, let's say maybe you're going to change some Terrform definition and you're going to push it to Git. To me, that's on the same level as application development that's in my head, that's code. Anything that is the human writes and it's readable by a machine, that's writing code. Now, I know that some people will disagree with me and say, writing YAML is not code. But my definition of code is it's human, it's written by humans, readable by machines.

Darin Pope 14:37
If you as a human have to care about how many spaces you're typing in, its code.

Viktor Farcic 14:42

Darin Pope 14:43
So you went away. You're pulling the marketing thing on me.

Viktor Farcic 14:47

Darin Pope 14:47
So the number one thing that Jenkins X solves is codifying good practices.

Viktor Farcic 14:54

Darin Pope 14:55
Is that the only thing that it solves?

Viktor Farcic 14:57
I think it is. Maybe you can run it. Am I missing something but except that I would say codifying best practices that envelop the whole lifecycle of your application?

Darin Pope 15:07
You said best. You meant good. But that's

Viktor Farcic 15:09
Oh, I'm going to put $1 in the swear jar.

Darin Pope 15:15
Best is just like things where it's a swear word now, right?

Viktor Farcic 15:17
I think that we had a discussion about that also before, not best practices, but it's not I don't know, necessarily have anything against best practices. What I do have against is how best practices term is effectively used. And I see that best practices is something that somebody says it's the best practice and everybody must use that forever and ever. It's like, one of the 10 commandments.

Darin Pope 15:44
Best is time bound.

Viktor Farcic 15:45
Yes. So

Darin Pope 15:47
or it should be time bound.

Viktor Farcic 15:49
Best practice is a good term if I ignore how people use it, but I'm saying good practice, mostly to distinguish.

Darin Pope 15:56
So a good practice means it's always good. It's sort of evergreen. It's doubtful that a good practice will ever be go bad. And it's doubtful that a best practice will become a worst practice

Viktor Farcic 16:09
Good practices as in when I say that my daughter is good. What I really mean is that she was good, good, good child behaving well today and tomorrow we'll see. We don't know.

Darin Pope 16:20
well, okay, I'll give something not so much in that direction but my daughter used to fence and she came in first at an event at a national event. She was 12. She's now 27. So I can no longer say she is the best fencer today. She was the best fencer that day, but 15 years later, she's not the best fencer anymore.

Viktor Farcic 16:47
Exactly. The only difference is that it takes like approximately a couple of weeks until a good practice in Kubernetes it stays good practice or best practice or whatever you want to call it.

Darin Pope 16:58
A best practice today does not guarantee that it's a best practice tomorrow. It may be, but it may not be. Whereas a good practice probably will always be a reasonably good practice.

Viktor Farcic 17:08
And also, no matter how you call it, it always depends on the context. Like here, I tend to say Kubernetes is the best thing that happened after peanut butter. But in reality, it's not the best solution for everything. It just fulfills certain use cases, certain needs. And tomorrow, it might fulfill other needs, or none of those at all. And the same thing with best practices. It's not a best practice to write getter setters, it's best practice to write getter setters in Java, when there are certain conditions in your application does this or that right? Not by default. Nothing is good by default.

Darin Pope 17:45
Well, unless you're getting paid by lines of code you write then getters and setters are wonderful.

Viktor Farcic 17:49
Oh, yeah.

Darin Pope 17:50
If you're not getting paid by lines of code you're writing then. I can't remember what the Java thing was that generates all the getters and setters for you automatically. There's some magic annotation that'll do it for you, but that's not what we're talking about

Viktor Farcic 18:01
Depends when you count lines of code to count comments?

Darin Pope 18:05
of course

Viktor Farcic 18:06
oh, then it easy it's easy to get well paid. Is that why Perl developers are complaining that they're not paid enough? Because they can write the whole application in two lines?

Darin Pope 18:16
In two lines? Yeah. They're really long lines, but they can be written in two lines. Okay. So let's boil it down, which problems was probably overstating it. The primary problem, there's other things that Jenkins X solves. But the primary problem that Jenkins X solves is that it provides good practices, codified good practices, what was the finish of the sentence of that,

Viktor Farcic 18:43
for the full lifecycle of your application,

Darin Pope 18:45
the full lifecycle of the application. If you don't like how those practices are, great, modify it, massage it a little bit to get it to where you want it. If you still don't like that, fork, how would you do that? Would you fork that process to come up with your own? What? What's the proper process for that? If you don't like the good practice that was presented to you, how would you fix it?

Viktor Farcic 19:14
If I would describe it as a collection of good practices, right, kind of now you do this. Now you do that. Now you do this and stuff like that. And you use this tool and that tool and stuff like that. So it's a collection of many things for if you can imagine the full lifecycle of something. It's a big thing, right? Now, if you say I want to change those two things, or those two parts of the process and stuff like that, so let's say if you want to change 10% 15% 20%, then yes, you just change the files because they're all in Git and and then different things happen. That's fine. Now, if you say, look, I disagree with 90% of those good practices or opinionated way to do stuff, then just don't don't go there at all. Jenkins for example, and for those the same like Bamboo and TeamCity and GitLab, they can do many, many different things. And they're good for many, many, many different things. And we might argue that it's not as the best in any single one of those things, but it's good at many, many things. It's very generalistic tools. And that's great because it gives you freedom to do almost anything you want. Jenkins X? No. Jenkins X is opinionated and like you, if you don't run Kubernetes, then go away. If you don't like GitOps, if you think that that's a silly idea, that's non negotiable, also go away. Now, if you don't like, let's say, Kaniko for building images, then you change it to build with something else and that's okay. So it really depends on some are non negotiable practices, while others are just opinion. Look, I like Nexus more than Artifactory right? So Nexus comes by default, you want Artifactory? Go for it. I don't really care. It's an easy change.

Darin Pope 21:01
Have you said your peace?

Viktor Farcic 21:03
I said my peace. Yes I did.

Darin Pope 21:11
Anything else you want to say about Jenkins X problem solving? Is it Turing complete?

Viktor Farcic 21:18
Not yet.

Darin Pope 21:23
Is anything ever truly Turing complete?

Darin Pope 21:27
We probably wouldn't know if it is.

Darin Pope 21:30
Exactly. It's like shooting an arrow and halfing the distance. The arrow will never reach its intended target.

Viktor Farcic 21:39

Darin Pope 21:41
I think that's a great way to sort of wrap it up for today. If you disagree with us, that's okay. We welcome disagreement.

Viktor Farcic 21:51

Darin Pope 21:52
Go listen to the episode with Joost. We, we politely disagreed. He politely disagreed with us. We politely disagreed amongst all three of us, I believe when it's all said and done,

Viktor Farcic 22:03
but it's unfair. So this is a complaint that I'm putting to ourselves. We did find a couple of people disagreeing with us and joining us in the show, but none of them ended up disagreeing with us at the end, as a whole. So we need to find somebody who really hates our guts, who really thinks that we're talking to completely if you are such a person, contact us

Darin Pope 22:29
send Viktor a message on Twitter. Better yet, join the Slack workspace. In the show notes there's a link down to the Slack workspace.

Viktor Farcic 22:38
Yes, because I really I think that this situation with confinement and Corona is making me more aggressive and I'm looking I cannot yell at my family and and Darin is such a calm person. So I need to yell at somebody. So come to the show so that I can yell at you because I really disagree.

Darin Pope 23:00
86753 Okay, so that's a different story. If you are listening right now via Apple Podcast, please subscribe and go ahead and leave a rating and review. All of our contact information including Twitter and LinkedIn, seriously reach out talk to us, we'd love to have you on. It can be found at And if you'd like to be notified by email anytime a new episode is released, you can sign up at The signup form is at the top of every page. And there are the links to the Slack workspace, the Voxer account and how to leave a review in the description of this episode. If you want to see really chaotic things happen, at least in our head, they're chaotic, join us for our live streams. If you haven't subscribed to our YouTube channel yet, go ahead and do that as well. The link to the YouTube channel is down below in the show notes as well. That is completely live unscripted typically and unedited, most importantly, unedited.

Viktor Farcic 24:04
Are you saying that this is scripted?

Darin Pope 24:05
It's scripted to the point that we actually had an idea of what we're going to talk about today. But that's about as a scripted as it got. We're never fully scripted. Except at the close here, because if you listen to the ending, that part is reasonably scripted. So Jenkins X, if you haven't tried it out, try it out. If you're let me rephrase that. If you're running Kubernetes workloads, and you're trying to find a good CI/CD solution, I'm making sure I'm still saying the right things here. Because I work on the other side of the house. If you're if you're building Kubernetes workloads, and you don't have a CI/CD solution yet, consider Jenkins X.

Viktor Farcic 24:48

Darin Pope 24:50
that's a that's a reasonable statement. Right?

Viktor Farcic 24:53
It's free. It's open source. Just try it.

Darin Pope 24:56
Just try it. If you don't like it, please let Viktor know.

Viktor Farcic 25:01
Yes. Then I can fix it.

Darin Pope 25:06
then you can fix it.

Viktor Farcic 25:07
Or I can promise I'm gonna fix it and not fix it. But

Darin Pope 25:13
you wouldn't do that. Thanks again for listening to episode number 60 of DevOps Paradox