Jonathan
00:00:00.000
the pace of change has increased. Spring Boot 1 was API stable for, I think, seven years, like no backwards incompatible change, seven years. And now it's, I think every three months. that's not similar in terms of pace. And so, if I'm, you know, go again to one of these, large banking customers and I have 20 or 30,000 applications written on that framework, that's a significant pile of work I have to undertake at a much like higher frequency than I used to. is DevOps Paradox, episode number 301. Exploring open rewrite and the future of code modernization.
Darin
00:01:40.035
Modern. But in case you can't say that, then. The name OpenRewrite might sound more familiar. Jonathan, how are you doing today?
Darin
00:01:50.455
Good. Now for people that don't know what Open Rewrite is, and we'll talk about Modern later along with a pretty cool event that's coming up in May of 2025 in beautiful Miami, Florida. Now, let me tell you, before we even get there, Miami in May is great because all the spring breaks are over. Colleges might just be now getting out. So when it, when it's happening, it's like the week to not have everybody on your beach.
Jonathan
00:02:19.200
It's warm enough. You can get in the ocean and still feel comfortable. It's great.
Darin
00:02:24.482
everything we talk about today, you can learn more about coming up in May. But what is OpenRewrite in case people haven't heard of it yet?
Jonathan
00:02:31.534
OpenRewrite is a large scale refactoring, software refactoring solution. And the main pillars of use cases here are application modernization, which encompasses language level updates, framework version updates, things of that sort. Security vulnerability repair is another one. Code quality, repair of the sort of things you'd find with like a SonarQ by fixing those is also another one. so in general, anything that you need to change in your code, but not just in one place, but potentially across many, many repositories, that's what it's for.
Jonathan
00:03:07.937
It started out and is still strongest in the JVM ecosystem. So in Java, we expanded from there into YAML, XML properties, Terraform, HCL, et cetera, all the kind of infrastructures, code stuff. more recently added parsers for JavaScript, Python, uh, C sharp. And we do have a, parser for COBOL. Copybooks, JCL, and about 31 related ZOS technologies as well. So kind of broad coverage of, I hope the most, some of the most frequently used languages, at least not all of them, but
Darin
00:03:41.936
Working with Java, now people that listen know that I work with the Jenkins project. The Jenkins project makes Strong use of OpenRewrite. So thank you, number one. but how did OpenRewrite even come about? what's its origin story? Did it come from Krypton or was it bitten by a spider? I don't know
Jonathan
00:04:03.141
It came from a special pressure cooker, which was a Netflix engineering tools. About 10 years ago, I was working on engineering tools. At that time, Netflix had a fairly small number of engineers, around 600, 700, something of that sort of thing. But,
Jonathan
00:04:25.966
this is, relative to Facebook about, you know, which is 15, 000 or something at the same time. So it's, but I see what you mean. Yeah, it can be a lot depending on the
Darin
00:04:34.231
yeah, because to me, six or 700 is bigger than most companies I've ever worked for. So that's just engineering alone. Okay.
Jonathan
00:04:41.696
That's true. That's fair. That's fair. Yeah. So no worries. Yeah. And so Netflix, you know, at the time, in addition to being one of the progenitors of microservices, there's microservices also had what they called this freedom and responsibility culture, which basically, for the sake of this conversation means that central teams like ourselves, like, you know, build tools or security teams couldn't impose constraints on what product engineers did. And so at the same time, we were responsible for, to some extent, helping the organization move forward from Java 6 to 7 and try to kill off this logging library that, we didn't want to maintain anymore. And so, you know, other sorts of things in the code in mass, but we couldn't say on a particular day, if you're not there yet, we're going to break your build, we're just going to turn on some gate and, you know, you're, you're not going to be able to build anymore because that would be a violation of that freedom and responsibility culture. And so confronted with this problem of how do we get the organization to move forward, I started, I know, in the same position that I think a lot of people do, which is trying to provide context to product teams of where they are right now. So kind of like early internal developer portal, IDP sort of stuff, like dashboards and materials that would, that would help folks understand where they were relative to the organizational goals. Okay. Thanks. And we did workshops and we, you know, uh, try to help people individually. And it just seemed like no matter what we did, it resulted in approximately no action, people wouldn't do anything. And so as I would talk to, to, you know, different engineers and ask the question, what could I do that would make you move forward on this company wide initiative, they would say, do it for me, otherwise I got something else to do, you know, it was kind of like, it was somewhat sarcastic, but. and so that's where, this sort of automation had to come about.
Darin
00:06:38.105
It's interesting that you use the word automation. I mean, I see it as automation, but also I don't see it as automation.
Darin
00:06:47.221
To me, it's almost like a really extremely smart and purpose built GREP for something or SED. SED, AWK, and GREP all coming together just in a, in a nice little, in the phrasing that's used within the OpenRewrite project is recipes, which makes sense and you can compose recipes together to have. You don't call it a meal, but that's the way I think about it is I can do a whole swath of things. I don't have to do everything little by little.
Jonathan
00:07:16.081
That's right. And, I'll be honest, we gave find and replace simple, said, um, I'll grab a pretty fair shake and, you know, attempted to do that logging library migration I was mentioning exclusively with that. it got very difficult. Obviously, the more you get to a point where it's crossing multiple lines and you've got nested method calls and so forth, and it's just, it's just too hard to make the pattern fit. moving up to. Abstract syntax tree level is kind of the next level. You go up from the text of code, you go up to abstract syntax tree, which is the, first thing a compiler does. It, it tokenizes and builds a tree out of the syntax that it's seen to reason about it. That level wasn't sufficient either, because while I could accurately identify a method call called info, I couldn't tell you whether the logger field that that info method was being called on was in fact. A Blitz4j logger internally, or if it was an SL4j logger, a Log4j logger. because the actual method signatures can look very similar. and so that's the abstract syntax tree wound up being surprisingly insufficient for even the simple transformations we were, we're trying to do.
Jonathan
00:08:32.206
it's certainly an improvement from, uh, from text. It's definitely a step up in understanding. But yeah, it's in many ways still a poor representation of what's in that code.
Darin
00:08:44.972
So what was the next big step? Because you found that, okay, basic search and replace, right? That's, that's where you started. It's like, how can, how can we safely do that? Right. Then you said, okay, how much further can we go with AST? Then you hit the brick wall again.
Jonathan
00:09:01.327
Well, that specific problem, and it was a silly problem. You know, the first one is just trying to kill off this internal logging library called Blitz4j. Blitz4j. It looked very similar to SO4J. Slightly different parameterization, slightly different order of arguments, but, pretty much the same otherwise. And, just, we didn't want to maintain this anymore. Why? You know, why? There's no reason to have a proprietary logging library like that. we just couldn't do it. I couldn't do it with find and replace, and I couldn't do it with AST only. trying to pursue the minimum. necessary information in this information representation to accomplish this change. I layered in one layer of what we call type attribution, which is the information about what these symbols represent. So for example, when I see a logger field name or variable name, I need to know what type that logger represents. And what I didn't need quite yet was to say. What are all of its super interfaces and super classes and so forth, all the way up to type hierarchy. I just needed to know kind of immediately what the, direct type was. And so, so how do you get that kind of information? You could go a couple of different ways. I could build An AST from TreeSitter, from Antelar, or something like that, that takes a text of code and turns it into an abstract syntax tree, and then tried us to create the algorithm that added the symbolic information. Like when I see this field, what type is it? Or I could decide to just programmatically exercise the language compiler itself. So the Java, Java compiler is written in Java. It's just a, it's just Java library able to ultimately, And I could try to guide it through its first few phases of compilation, through producing an AST, through going and solving for all the symbols on that AST, pause it, and then pull all the information out of that internal representation in the compiler, and act on that. And that's the path I chose because I didn't want to reinvent all the symbol solving that's inherent to the compiler. I thought that's a complex problem in its own right, and it is.
Darin
00:11:09.966
So it sounds like it sounds like you were able to do that and maybe a weekend. That's, that's what it feels like, right? That's. That's all it took.
Jonathan
00:11:16.476
Close. It was, this is one of these Christmas projects that I often do. It was one of these, like, you know, late December, early January, everybody's kind of out. And you kind of get time to get heads down and just, just soar for a while. And a lot of the basic, elements of Open Rewrite were created then, you know, over that Christmas and I think 2016.
Darin
00:11:38.401
that still wasn't Open Rewrite. That was still you were solving for that problem within Netflix. How did that come out of Netflix to become Open Rewrite?
Jonathan
00:11:48.751
Luckily, Netflix was very permissive about open sourcing tools internally because, and they were famous for this with Eureka and Ribbon and Hystrix. And, you know, the, kind of the whole Netflix OSS stack of microservice development that was popularized. this was true for our build tool plugins as well. The Nebula suite and the Gradle plugins. Spinnaker, continuous delivery solution. and so OpenRED from the beginning was just Apache licensed there. The feeling was that we're not done when we're done with Blitz4j, that there's gonna be work like this that is gonna keep coming up over and over and over and over and over again. And so it was worth investing
Darin
00:12:27.745
My favorite was Hystrix. I'll just leave it there. That was Feature Flags before Feature Flags was cool.
Jonathan
00:12:33.450
yeah, in many ways. Yeah, super cool visualizations of that internally. At some point, kind of sense that this problem is very broad. it's not just Blitz4J and SL4J. It's, we're doing Java version updates. It has a lot of different, steps to it. there's just any modernization effort we want to, pursue could be solved with something like this, this, these programmatic recipes. And so I remember going to a skip level meeting with the VP of engineering, Yuri Israileski at the time. He's since moved on, I think, to Clickhouse. He heads product over at Clickhouse these days, but Yuri just, you know, I was ready to advocate for building a team around open rewrite or, you know, just mass refactoring thing inside of Netflix. And his immediate answer, and wisely, was just no. it just felt like it was too big of a problem to live inside a company like Netflix, I think that's why. Uber, when they worked on Piranha for, refactoring, a little bit different approach but similar goal, Google working on Refaster, Facebook working on Codemods or JSCodeshift. They all kind of got so far and then they gave up on those efforts because it could really take on a life of its own inside a company like that.
Darin
00:13:50.056
And thankfully for you, they did stop because that gives you the ability to go ahead and keep going with open rewrite.
Jonathan
00:13:56.591
It was one step in a journey that eventually led to the founding of Madarin, and it wasn't immediate. So, I didn't take this courageous step out of Netflix and say, okay, well, I'm going to go do this as a separate company. I actually took a detour and went to the Spring team and founded a project called Micrometer and started working on Spinnaker. And so just kind of in a different space of the developer journey altogether. And it was there that I was working with. And, uh, you know, I was working with Madarin co founder, Olga Kunzich, on Spinnaker, in large enterprise environments, trying to teach advanced continuous delivery, automated canary analysis, this sort of thing. And I kept seeing the same problem. They would say, well, I'd love to do automated canary analysis, but talk to me in a year when I'm done moving Spring Boot one to two or Java six to seven or whatever the emergency of the day was. Uh, and we just, you know, what was sort of startling is how repeatable the problem was. across, it didn't matter what vertical, like, you know, whether it's financial services, retail, whatever, or size of company. And it's really because we're all building on the same third party and OSS substrate to some extent. We're selecting, you know, from the same component bins and that stuff evolves at its own rate. And so it's either you keep up or, potentially your app's not functional at some point, or it's got some sort of, you know, Operational issue or security issue later on.
Darin
00:15:19.052
Well, the story that you just told there, Spring Boot 1 to 2, or any, or thinking JUnit 3 to 4 to now, in these worlds, 5. Or, even worse, Java 11 to Java 17 to Java 21 and soon to be Java 25. Right, just to keep LTSs. Don't, don't take the ones in between. Stay with the LTSs.
Darin
00:15:40.942
Just do, just do that. and this, and we can talk about Java a lot because Viktor isn't here right now, cause he would just be telling me, well, Java is not a real language. Yeah. It's, it's not a real language. It's it's, it's
Jonathan
00:15:51.110
And as far as I can tell, tell Viktor is not a real person. So see that works out, right? You know,
Darin
00:15:55.610
Yeah, it's okay. Um, I'll, I'll take advantage of that while he's not here. Uh, but let's, let's go back to my JUnit example, right? JUnit 3 to JUnit 4 is not trivial.
Darin
00:16:22.270
Is that just a downside? Okay. Let's we've been happy with Java. Let's bash on Java. Is that just the way it is? We just have to deal with it and go through every wall to get to where we want to go. Is there no way to short? I mean, I guess sometimes there's a way to shortcut it, but I've never seen it
Darin
00:16:43.435
The you have to that pretty much, you always have to go incrementally. Yeah. Which, I mean, sometimes you can make a jump. Like I said, that's not all the time are people trying to solve for the jumps here? Here's what I think I'll, I'll, I'll put in my 2 cents on this. People have been running on, they're still running Java six and they wanna go straight to Java 21. You're not gonna do that in one hop, realistically. What do you think? It depends.
Darin
00:17:13.452
So what, what would be the depend good and depend bad? Like if I want to change all my, if I want to go to streams in one pass from six to 21, can I do that? Could I rewrite
Jonathan
00:17:26.962
Oh, I wouldn't rec I wouldn't recommend that anyway, but, but I know, I know it's a bit of a, I'm, you know, attacking the example. Yeah. Yeah, it's, it's, I was, interestingly, I was talking to, a banking executive a couple months ago and he was talking about wanting to fix one particular vulnerability, that regulators were hounding him on. And he said, you know, I just need to update the version of this thing. And I thought, well, this thing is fairly easy. It's got 3, 000 applications, but nevertheless. But the catch here is that the minimum fix version of this dependency that fixes the vulnerability has a baseline of Java 17, and I'm on 8. I thought, okay, you know, slightly worse, but we have a Java 8 to 17 recipe, we can do that. He said, ah, but the additional rub is that all 3, 000 of these apps are running on WebSphere with a maximum version of Java 8, because they're only entitled to a specific version of WebSphere, whose maximum version is Java 8. And so to even move them to 17, I have to move all these apps from WebSphere to Tomcat, which means replatforming entirely. And so then the question becomes, okay, well, I mean, this is getting like scarier by the second, right? You know, but I thought, well, what are we talking about here? This is running on WebSphere. Like, is there, are there a bunch of EJBs? Is it like what, you know? Nope, just straight circuits. not a single EGB, just straight up servlets. So it, it could just as easily be running on Tomcat. It's just, it just happened to be running on WebSphere. And so there are circumstances like that. And sometimes across substantial business units where there's not a lot that needs to change, you know? So the one thing that, that did need to change in addition to the like regular Java 8 to 17 sort of move was that it was using WebSphere MQ and needed to move to something like RabbitMQ. But even then, like. I can definitely execute that with recipes. So, it's, uh, would require testing of course, but, the programmatic changes can definitely be done. So it's, yeah, it very much depends on the technologies. How much of the technology surface area are you using? I think.
Darin
00:19:23.022
Well, I think the thing you peeled back there is a tale as old as time. We can make a code change, but just making the code change doesn't mean that's all that's got to be done. Because as you said, they would have to re platform from WebSphere to fill in the blank. there was probably other transitives that came along that made things that you didn't call out. that's sort of our, I could say that's part of our, Living in the JVM ecosystem, but, you know, dealing with transitives. We're used to it, but I think it's even worse in JavaScript land.
Jonathan
00:19:55.413
Yeah, well, I mean, yeah, it used to be complete class path isolation or componentization there. And now it's announced. I mean, that has its own downsides, but, not anymore. Now there's conflict resolution. So welcome to our hell, where everyone ultimately winds up, sooner or later. But, yeah, it's definitely nasty, I think, and it's only in different ecosystems. I think we focus a lot of our time on the social engineering aspects, which is what you're kind of getting into here. if you're new to the space of mass scale auto remediation, you're sort of starry eyed new to this, maybe you're a new company in this, or you're new to, you're just, you're just kind of ideating on how would I do migration engineering, it's easy to think I need automation to make the code change, and then I need some system to do mass pull request issuance. once provided with that system to do mass pull requests, kind of build it and they will come, right? Like issue the pull requests and they'll be received on the other side. at the, you know, outset of this effort at Netflix, I found that not to be true. I think people perceive pull requests, coming from external teams, often like unwelcome advice coming from an in law, sorry for in laws out there. But it's like, it could be great advice. But, it feels unwelcome. And so you just look for a reason to reject it. And curiously, so we call that push based change when, a central team is trying to push change horizontally across the whole business at once, that does not tend to yield high merge rates. The number we've seen time and time again, and I don't know why this is the number, but is around 30%. You'll get about 30 percent out of that push based mass pull request issuance. And then there's the other side of this, which is, so you have to create mechanisms that are pull based. In other words, provide enough context for a team to see that there's a change to be done and then provide the exact same automation that would have been used to create that pull request. But put the button to create the pull request in the product team's hands and the success rate is much higher. It's just a human nature, sort of social engineering problem of like, if I open the change, I'm more likely to merge it.
Darin
00:22:09.193
That just seems so bizarre to me. But again, take a look at every open source project and see how many open PRs there are, and I will tell you how good that project is because if there's enough and they're, they're sort of flowing reasonably, then it's probably okay, but if it's stacked up and you've got PR setting from three years ago, is that a project you really want to be using, or is that a component that you want to be using?
Jonathan
00:22:33.603
It certainly discourages some degree of contribution, right? Cause yeah. Yeah. It's hard to be perfect on that too. As an open source, Depends on how much, how many resources you have, but yeah, you're right. It, it, it can be difficult if you, if you don't keep up with it.
Darin
00:22:46.995
So let's talk about the recipes real quick, or maybe not real quick, because that's the sort of the real magic to this, I think, because you, And again, Modern has a different thing. We'll, we'll talk a little bit more about that later, but just if I was wanting to use, like if I'm willing to do a quick start, I've got a little open source project and I want to use open rewrite and I'm wanting to move from, let's, let's use two recipes. I want to move from eight to 17, Java eight to 17. And I'm also wanting to change all of my JUnit tests. I'll use that again from three to four. Right. Cause I'm, I'm, I'm just old. With this, and there might be Spring Boot in there too, or something, right? Just one of the frameworks. Do I have to chase down, do I have to write all that? So what do I do? I mean, it's like, how do I get started? Where do I even start? How do I even start thinking about it?
Jonathan
00:23:36.056
if you look at docs. openrare. org, you'll see the reference for the available recipe catalog, which is, you know, recipes are programs that do individual units of transformation. And those could be as big as Java 8 to 7, 17, or it could be as small as change just this method name on a particular type or something like that. So, you know, there's thousands of these recipes out there. The ones you mentioned Java 8 to 17, JUnit 3 to 4. and Spring Boot migrations are all available out there in one form or another. so you just select the recipe. You configure, if this is, you're doing this on one repo, you go to the, you apply a Maven plugin or a Gradle plugin, to your project. you configure it with the recipe and then you run a build tool command, a Maven or Gradle command that runs the recipe. It will produce patch files or diffs. For all the changes. And then you can apply those to the repository. And so it's basically pre staged the changes for you. That's the, goal. And then from that point, you know, you choose to commit them or not.
Darin
00:24:42.235
thing in that, you sort of glossed over it is just renaming a method, like let's say, let's say I messed up and I didn't camel case correctly, just because I did something silly. There's not necessarily a recipe for that method, but there are generic recipes that are easy to configure that I can just say from to. Those aren't the
Darin
00:25:04.600
but it's something in that. Because we did that on some projects within the Jenkins project to where we just needed to do a couple of small things like that. But it's like, okay, I don't need a big recipe. Like standalone is like, there's already just some utilities, like a little screwdriver, a little hammer, right? That will do the one specific thing without me having to write something. And that's the key thing is when I've looked at some of the recipes, I'm head scratching for a very long time. Trying to understand what some of, talking about the Spring Boot changes, right? And you know, okay, what's really going on? Because those are complex, but then we're also talking about, we can, you don't have to nickel and dime each one of those recipes. You can stack recipes together. So it's like, okay, again, the way I think normally is, okay, let me do this one thing. Then I'll go do this other thing, then I'll go do this other thing, and however many other little things I need to do. That's like, okay, I might do that in another pass one more time. It's like, okay, yeah, that worked. Then the great thing, I call it a menu. That's, I don't remember what you got. It's, but it's a composition of all those recipes come together, and then you just apply that composition. And it does, goes off and does all the things, which is super cool.
Jonathan
00:26:14.633
That's right. And what I like about stacking them compositionally like that is you can choose to take the top level thing, like Java 8 to 17, and just apply it all at once, or if understanding all the constituent changes in that is overwhelming to you, you can look at one of the sub recipes, like Python. You know, and just apply that. so it'll be fewer changes for you to think about at once and you get those through and then you go on, go on and do the next set of changes. We see that very commonly be an accelerant to pull base change to actually get things through is how much it's really depends on the risk tolerance of the business unit receiving the change. If I'm trying to get everybody to Spring Boot 3.3 Well, to get 3.3 you have to get to 3.2 which means you have to get to 3.1 which means you have to get to 3.0 which means you have to get to Java 17, which means you have to get to 11, you know, and so do I ask them to do it all at once if the app is, say, like on Spring Boot 2.4 or Java 8? No, in many cases, it's easier for them to just get to Java 17 first and stay on Spring Boot 2.4 get that to production, rest with it a little bit, be comfortable with the operationalization of that, come back and rerun the recipe that includes Java 17, which now won't do anything because it's already been done and do the rest of the change. And oftentimes it gets broken down even further than that. I think a really great example of this. These are organizations that don't, they use Maven say, and they don't have Maven wrappers checked into all their repositories. Now whether you can even move to Java 17 entirely depends on what CI server the build winds up on. Because if the CI server has a particular version of Maven on its path variable, version is not Maven 3. 3 or whatever it is that you can develop with Java 17. You could change all the code to Java 17. It's going to break when it lands in CI. So, a good like first step is like, let's just get Maven wrapper 3, 4, 3, 5, or whatever checked in everywhere. None of the code changed. None of the runtime changed. No risk. if it built yesterday, it builds today, but let's just get that done. That gets me used to the idea of mass committing something. and sometimes it has organizational dynamics. How do I attach a JIRA issue to it or GitHub issues issue to it? Or do I get approvers on mass, you know, and, and so you kind of work through that process. Then you go back and you do the slightly more complicated thing. And all the while you're kind of building up trust in this muscle of, Oh yeah, I can actually like, I can actually do things. Horizontally across my whole team, as opposed to just one at a time.
Darin
00:28:48.219
two things out of that. Number one, yes, kids, there was a time before Maven Wrapper. It was a bad time.
Darin
00:28:57.259
It does still exist. number two, do one thing at a time until you trust it and then keep stack. The one thing I hadn't thought about is, okay, maybe I start with my composition from day one, but the only thing in the composition day one is the one thing. Right. And then when I'm ready to add the next thing, then I just, just tack it on the bottom and sequence does matter by the way,
Darin
00:29:24.996
But then you were talking about to all the places. Now it seems like I would have to have all the repos on my machine in order to do that. Right. But I have a, I have a feeling that's where Modern starts to come into play.
Jonathan
00:29:40.931
That's true. And maybe I'm kind of, confusing these two a bit, if we go back to the open rewrite experience before it was out of build tool, plug into the repository, run a build tool command, of course, how you configure the build tool is dependent on whether it's Maven or Gradle, and then, you have to run that build with a specific version of Java, say, or, you know, depending on the version of the project and its starting point. So it gets very complicated. yeah. Madarin is meant to just exercise that same recipe. But across a set of repositories and potentially hundreds, thousands, tens of thousands of them. there's one other main distinction there, which is OpenRewrite, basically what's happening under the covers when you, you know, So the way we run a recipe is it's parsing the code into what we call the lossless semantic tree. It's AST plus all that type attribution information we were talking about earlier. Plus all the formatting of the original project is important as well to preserve style. But we keep all of that in memory. And then the recipe runs and it makes changes to that tree or extracts data from that semantic tree. And then those changes get, materialized back as source code to make the change. the process of bringing code from its textual representation on disk to loss of semantic tree and memory is by far the most Time and, resource intensive process of the whole recipe run. The recipe run itself is negligible in comparison. one of the differentiators really between OpenRewrite and Modarin is that those loss of semantic trees that are in memory in OpenRewrite open source, we are able to serialize to disk in the form of artifacts, in Modarin, and so then you're able to basically in, in sort of an offline process. Create LST artifacts for whole business units and then pull them and operate on them as whole business units, as opposed to on a per repository basis.
Darin
00:31:41.977
Which is good if you're trying to basically have the easy button, we'll give Staples their credit on that.
Darin
00:31:53.110
modern. We'll get you. Yeah. Well, but I guess the way I think about it is. Open Rewrite is the battery and the clips and the button itself and the casing and everything else. And where Modern is just, here, here's the button, press the button. Not, I'm, I'm, I'm extrapolating too far.
Darin
00:32:12.610
all the pieces. put together. Yeah, But again, just Open Rewrite itself Open source, Apache licensed, everything's sweet to go. And it does make a difference. I can speak firsthand on that. Now getting my head around it at first was, but in,
Darin
00:32:32.795
in, in full disclosure here, Jonathan and I actually met in real life. That doesn't happen very often. Uh, we met in, at the Dallas Java user group in November of 2024. And. We had a chance to talk there and it's like, Oh yeah, this, we've done this. now we're now we're finally talking here. It's interesting though. I'm, I'm looking at what OpenRewrite is today and it makes sense. There's a big community around it. It's not just Jonathan writing all these recipes.
Darin
00:33:17.345
Yeah. that's probably, which is more fun, writing the recipes or wrangling, probably sometimes. Um,
Darin
00:33:26.445
one makes you appreciate the other, probably. where's OpenRewrite headed in the next. Three to five years, because if we think about it, Java has been around now for, we're coming up on 30 years in a year, roughly. Based on what you're seeing, you're still seeing WebSphere with a minimum requirement of Java 11 today in 2020. Oh, sorry. Java eight. Yes. Max. Yeah, maximum eight of Java, Java eight. and this is 2025. Now we're, we're not pre-recording this one too early. and I see that as well.
Darin
00:34:01.988
Java's gonna be around. Java is the new cobol. Yeah. I mean, cobol, well, COBOL will probably outlast Java, but that's a different conversation.
Darin
00:34:09.979
are you going to keep up with this? Because, okay, let's, let's play out this scenario. I'm losing track of when, everything comes in, right? So let's say there's a new Spring Boot that's going to come out in a month. How long is it going to take, not you, but the community around it to actually have a recipe to get you up to whatever that next Spring Boot level is?
Jonathan
00:34:33.314
Well, maybe this goes to the market in general and what you can see OpenRewrite is, is a two sided marketplace, which is there's recipe producers and there's recipe consumers. And so like any marketplace, it's how do I get the appropriate population in each? Airbnb wouldn't be anything without any houses nor people willing to rent those houses. And so one interesting dynamic here is that, well, ideally I want the framework authors, the ones making the break and change. To be the ones that write the recipe that fix their downstream consumers. After all, they're the ones with the most context in their mind of why they made the change and what the correlated thing was supposed to be. They document this oftentimes in migration guides. Why not just write the recipe, along with the migration guide and really help me get there, get there. I can try to promote a, altruistic view of this, like, Oh, you, we should, you should, as a framework author, provide this, recipe so that your downstream consumer has an easier life. But I don't think I have to lean on altruism here because we work with some of the largest companies in the world. I think six of the top banks, ten banks in the world, you know, a lot of large insurers, some of the largest retailers in the world. What's interesting here is the power dynamic. is that there are customers of Broadcom that owns the Spring team, there are customers of Red Hat that owns Quarkus, there's customers of Oracle that owns Micronaut, etc, etc, etc. These, you know, most of these large frameworks exist somewhere, you know, in some, fit in some commercial entity. honestly, I mean, let's just be frank, they can put spend at risk back with those entities to say, You're breaking me constantly by using your platform, and I'm sick of it. I'm trying to build features forward for my business. You keep breaking me on the back end, so if you break it, you fix it. I don't expect this stuff to be public a lot, but we've actually seen this in very concrete ways, in some of our customers actually putting spend at risk. Back with those big vendors to say, you owe me her recipes. You know? And so I don't, feel like I'm not leaning on, some sort of like, you know, we all kind of, uh, agree that this is the best way to go, but I think that there's actually the right sort of dynamic to be self-reinforcing on this two sided marketplace.
Darin
00:36:45.813
As a quote unquote maintainer, I don't like the idea of having to write yet one more line of code,
Darin
00:36:53.873
but it makes sense because if I'm already writing a migration document, why not just, if people want it, it's like the migration document should, let me restate it, the recipe should be a codified version of the migration document,
Jonathan
00:37:08.559
That's right. it's codifying your knowledge Yeah. to be clear, I mean, I don't expect, I don't, I don't mean to say I expect every framework author to do this in every case. I just think, I believe in market forces here, and if you're a, a proponent of a major or you're, you're a proprietor of a major framework and there's competing major frameworks, well, I evaluate frameworks as a user on multiple. One of which is just the pleasure of working with it, you know, how well tested I think it is, how secure I think it is. And I mean, I think one of these dimensions I have to consider going forward is how easy is it going to be to keep up with changes in this framework going forward? That's a big reason people didn't select Scala, right? It was actually, like, it was known to be hard to keep up with changes to that over time. It's a reason why people have stayed with Java, because it's relatively easy to keep up with changes going forward, or it has been, historically. So, I just say I would add it as another, decision criteria, when selecting a framework that you use.
Jonathan
00:38:11.669
How well are they going to treat you? Yeah. I think you were hinting at something also before that I think is significant and maybe kind of flew under the radar, which is the pace of change has increased. Spring Boot 1 was API stable for, I think, seven years, like no backwards incompatible change, seven years. And now it's, I think every three months. that's not similar in terms of pace. And so, uh, you know, if I'm, if I'm, you know, go again to one of these, large banking customers and I have 20 or 30,000 applications written on that framework, that's a significant pile of work I have to undertake at a much at like higher frequency than I used to.
Darin
00:38:54.066
So where does AI come into play in all this? there's AI running underneath all of this, right? We've already proven that there's not at this point.
Jonathan
00:39:01.403
It's kind of in some ways the other way around and, but I do think that there is an interesting play here. so first of all, we've seen, you know, the emergence of things like Amazon KubeCode Transformer. Andy Jassy famously announced, Amazon Q Code Transformer saved something like 7000 years of effort internally. what was in the blog and not in his keynote was actually Amazon Q Code Transformer is built on top of OpenRewrite. So the engine that's actually doing the work under the cover is, OpenRewrite for that big Java migration. it makes it sound like it's AI on top, and there's certainly an AI verification loop on top of it, but the primary engine was a rule based one. Similarly, Microsoft GitHub Copilot Upgrade Assistant very transparently uses OpenRewrite to do its work. Broadcom Application Advisor very transparently uses OpenRewrite to do its work. IBM Watson Migration Assistant, I believe, uses OpenRewrite to do its work. So, the ones that are doing mass change lean back on a rule based system because the scale of change it can create is, you know, you can't have you can't have this potential for hallucination. On the other side of software development, which is net new code authorship, writing out net new code. That, potential for hallucination actually is not very, it's almost a feature, right? You know, it's the things that Copilot suggests are by definition, the things that my IDE intention actions were never good at. So it's, They're very complimentary. it accelerates my net new recipe development. It accelerates a longstanding trend towards industrialized software production, which started with a general availability of third party and open source stuff and great IDE tooling. I developed net new code a lot faster, but then it was on the shelf and how do I maintain it? And that's a problem that we haven't had in a sort of industry wide solution to for a long time. And so I see this like AI code authorship. Accelerating, putting the gas on net new code authorship. And we really do need to book in that with how are we maintaining it on the, on the opposite end.
Darin
00:41:03.788
So if people don't care about their old code, they can just have AI write everything. But if they care about the things that's actually making them money, they've got to maintain it, or do they? that's the tale as old as time as well. It's like, do I really need to maintain something that's 10 years old?
Jonathan
00:41:19.277
I think there's an old Joel Spolsky article. So for, and I mean, we're dragging up stuff from the past here, and I think this is almost 20 years old now, but Joel Spolsky for those, maybe newer listeners that don't know, you know, one of the, the founders or heads of, uh, Stack Overflow. he's written some great materials on software engineering over the years, and there's a paragraph somewhere, and I don't remember exactly, maybe we can link to it somewhere in the show notes, but about the cost of a line of debugged code in production, that was great, because it's like, you write a line of code, you operate it, ah, you figure out a couple things, you make some changes, you, you know, maybe it's performance, maybe it's functional. it doesn't happen all at once, right? code kind of like reaches a functional state over many iterations. so I think it's very difficult to just throw out everything, you know, because it's not just about kind of banging out lines of code. It's what's the cost of a debugged line of code. That takes time.
Darin
00:42:15.842
And then it's the other side of, this is a very prescient point for me right now. Uh, at the time of recording, git2. 48. 0. 1 is coming soon. Just came out. it broke some things. And of course, I'm, I'm a good person on my Mac. I update all my rest, you know, update, I keep everything updated. And now it broke things. So now I'm having to debug. And figure out, okay, do I fix, do I roll back? And yeah, well, that's when I'm, it
Jonathan
00:42:48.034
Yeah. It's always hurts when it does. I used to have a command, you know, a whole series of tooling. I lost it somewhere along the line called yippie ki yay, which is just, you know, you run yippie ki yay in the morning and starts updating a bunch of different types of things, you know, and go get a coffee or whatever, come back and we'll see if you're in a functional state or not.
Viktor
00:43:16.416
I mean, I'm of opinion that actually updating very frequently is the least fateful option.
Viktor
00:43:24.993
upgrading a single library. To the minor, next minor version or patch and then doing that thousand times is infinitely easier than waiting until I need to update thousand libraries.
Darin
00:43:46.016
in the long run, it's, it's, it's busy work. But it's not necessarily work because of course you have all your tests written, right?
Viktor
00:43:53.311
even without tests, right? If I update a single library to the next patch, and something goes wrong, I know what's wrong. I mean, I don't know exactly which line is wrong, but I know where to look. Right? This is what happened, right? This is what was updated. When multiple things are upgraded at once I can only scream. That's the only thing I can do. Kind of, to do, I mean, maybe not the only thing, but that's the first thing I do.
Viktor
00:44:30.171
Yeah, right. I feel that, and we see that in industry often, that what I'm about to say is not only related to upgrades, but in general. Kind of, hey, we, it's too expensive for us to do this today. We're going to do it tomorrow. And then you realize that actually tomorrow is not double expensive, but triple. And then you say, I'm going to do it day after tomorrow. Until you reach the point to say, there is nothing I can do. This mainframe is going to stay here forever.
Darin
00:45:05.088
inflation is real. So OpenRewrite can be found at openrewrite. org. We talked about Modern, I'm going to spell it, it's Modern, M O D E R N with an E on the end. So Modern, very French, dot I O, which is the, it's a SaaS offering that has, uh, It makes, we were talking about doing, so OpenRewrite, if you're just working on a single repository, OpenRewrite is your friend. If you're trying to manage a set of repositories or even a full organization, thinking GitHub organization, uh, you also support GitLab as well, I believe?
Jonathan
00:45:44.828
I've actually seen an organization with nine distinct version control systems. Operate in at once. Um, and so in that case, we'll operate on all of them.
Jonathan
00:45:54.760
Well, six of them are Bitbucket and three of them are GitHub. it's just multiple copies, different network segments.
Darin
00:46:01.115
Which is interesting because if there is a common library that all of those things use, I mean, you could pull it off just using open rewrite, but having that visibility to where, okay, I've wired everything in now, here's everything like we were talking about earlier, we want to go through and make sure we don't have anything named, okay, I'll use this because let's say we're wanting to rename every, every master to controller, we're going to rename slave to agent, right? Trying to, you know, I have friendly naming, being able to do that at not just an organization level, thinking product level, but like my organization level, which includes who knows what, and just having that visibility is like, and again, not automatically putting the PR, although it could, right? Because I could go and just fire those off. But again, maybe it's an engineering manager's go through and it's like, Oh yeah, these are good. These are good. These are good. Or whatever it is.
Darin
00:46:55.513
If you're interested in that, you can check out modern. io, but we're talking about Miami. Viktor, do you want to go to Miami? Could you hang out in Miami?
Darin
00:47:15.233
Modern is, you're sponsoring with the help of the team at NOF. Thank you. No Fluff, Just stuff. It's called CodeRemix. Even though we're saying Miami, it's actually going to be in, Viktor, you might like this. It's actually going to be at the Margarita Hollywood Beach Resort in Hollywood, Florida, which is north of Miami, actually north of Fort Lauderdale. if you can get into Fort Lauderdale, much easier airport to get in and out of than Miami, trust me. if you want to go, there is actually a code that you can use. To save $300 and it's, it's not one of our codes, so I'm, we're not getting anything on this. It's modern three. The number three VIP gets $300 off, and this is May 12th through the 14th. And if you've ever been to any of the No fluff conferences, there's a lot of the, the common people that you're gonna see there. A couple that I saw was, uh, PR Patel, Josh Long, uh, VIN Kat. Yeah. You name it all, all, all the names that you're used to. You can check that out@coderemix.ai, not io ai. Why ai? Come on. You're not doing AI here or are you
Jonathan
00:48:21.758
Yeah. Well, that's a whole nother subject and, and maybe worth like one or two minutes at the end. There is a little bit there. one more thing about this conference, I think the way we're trying to organize this, cause I'm a developer, I still like hands on development. You can't pull me away from it. We're doing three tracks. So there's a developer track. that's where the Venkat, Josh Wong, you know, Baruch, uh, you know, like you said, regular suspects, also some really new, interesting voices. I mean, I think we're inviting somebody from Patient in Columbia, Missouri, that's talking about a really new, interesting language called Rock. Um, that's one I'm really interested in, uh, going to. So developer, leader track. so we'll have, folks from, some of these, yeah, the largest banks, retailers in the world talking about how they're doing, uh, migration engineering like this. and other topics. And then the one I'm kind of most excited about is the HackTrack, which is going to be going on for three days. we're going to have different tables set up with particular, tooling and things, like, to play with. And so what we're asking attendees to do is come with your work laptops too, and like, as much stuff on them as possible. And get the, like, get the security guys or whatever to, like, get over that before you arrive. come with that stuff and, you know, we're going to do things. Like, so, for, like, recipes, it's like we're going to write new recipes with you, custom recipes. We're going to apply them. So the idea is, like, you come with some stuff and you go home with work done.
Viktor
00:49:50.107
Shouldn't people come with private laptops then? Instead of company laptops with VPNs and all those things that prevent you from doing anything useful?
Jonathan
00:49:59.412
I want company code. That's the main thing I want. Because we want to. The goal is to provide as much value, and so I've done a lot of workshops at NFJS conferences, DevNexus, et cetera. And one of the common problems I encounter is, like, there'll be people in the audience that say, Oh, I wish I had brought my work laptop. I could have done this Java 17 migration on. But so it's like, come with your stuff, get that security guy to approve it, because we want to, we want to do like real stuff, not, Just hypothetical things.
Darin
00:50:27.387
we are not lawyers. We are not advocating that you break any of your corporate policies at all. Get approval.
Darin
00:50:34.597
regardless. CL colonial repository first because. You don't want the computer connecting to the network. We'll download all your dependencies. Just, just think as if you're going to be on a plane for 12 hours, actually, uh, 72 hours, you're gonna be on a plane, a flight for 72 hours. Uh, and so you can work. And think about how much work, I mean, I don't know about you. I get probably most of my work done on an airplane. Maybe
Jonathan
00:51:05.477
time. Guaranteed focus time. So the one brief bit about AI, I think this is accidental, but because we built up this loss of semantic tree over time with all the information that compiler knows and all this stuff, because we were trying to write recipes to make changes, I think we came into possession of a data source that has information that's Below the text of the code. when you can do that across a whole, whole business unit, or it, it winds up being an ideal tool that you can, enable a large language model to use, to reason about code and to suggest change in a more accurate way. It's tempting for folks, I think, to say right now, models will get better. They'll get better. They'll get better. They'll get better. I think one of the key problems is that. Oftentimes they're operating on text or AST. It's just not enough data. It's just not enough information. Back to the original example we talked about at great length about Blitz4j to Log4j. Knowing that there's a field called Logger doesn't tell me whether it's Blitz4j or Log4j. I need to know what the compiler knows about that field in too many cases. Just last night, I put together what's a model context protocol or MCP server that's enabled with a bunch of open rate recipes that are acting as tools and bolted that into Cloud. And so from Cloud Desktop Client, I can ask a question about how this particular technique is used in thousands of repositories. what it does is it calls the tool, runs all that, collects that all up into, you know, open rewrite data table, provides that back to the model. And then it can further reason about what to do with that data. And so I think there's a lot of interesting use cases that are going to come out of this rich data set that we built for migrations or refactoring originally, but yeah, didn't anticipate this application.
Darin
00:53:03.982
If it can be a tool for AI, that's going to be the bot. I mean, based on what you were saying before, AWS, that's Basically how they wired it up. It was a tool. So just if you, if you're already using tools today, this is just another tool in your tool belt.
Darin
00:53:21.017
Well, all of Jonathan's contact information will be down in the episode description. I also have links down to that Spolsky article and the link. Well, I haven't found it yet. I know what you're talking about. I remember it, but I'll have to find it. Uh, and also the link off to CodeRemix with the code that will save you 300 up until a certain point, if you're listening to this after May 12th. It's over. So sorry. You will have missed out in Hollywood, Florida at Margaritaville.
Jonathan
00:53:49.757
Thank you. We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox. com contact. If you subscribe through Apple podcast, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox. com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.