Wesley
00:00:00.000
The way that I often will approach a problem is to first kind of seek out as much prior art as I can to just kind of get a foundation of like, especially in something like Go where I was a relatively new developer, right? Like probably other people have thought about this problem and have more knowledge and experience to bring to bear. So. Me following the examples of what other people have done is likely to end up with something better than me trying to just invent something that is going to be, basically Ruby poorly translated into Go, This is DevOps Paradox, episode number 303, how to develop a CLI in 2025.
Darin
00:01:31.998
Viktor, how much of your time each day do you think you spend at the command line running CLIs?
Viktor
00:01:38.850
how many hours there is in a day? 24 minus eight hours sleep. Let's say one hour for, let's make it two hours for food. That would make it, uh, 14 hours.
Darin
00:01:51.336
That sounds about right. 14 hours. Because. Can you get your job done today at all without going to the CLI?
Darin
00:02:06.021
Okay. Same amount of time. There you go. Okay. So on today's show, we have Wesley Berry on. Wesley, how you doing?
Darin
00:02:15.421
So we have Wesley on, he's from Anchor, and we'll talk about Anchor sort of throughout the show today. But Wesley had a CLIs. And it sounds like y'all went through a lot of pain and suffering. Trying to get your CLI built for Anchor. Is that a semi fair assessment? Just taking a look at the first intro to the blog post, it was like, oh, this doesn't seem to be a good story at all.
Wesley
00:02:45.161
Yeah, I mean, it was, uh, not as smooth as I might have hoped, prior to that I'd done some CLI work, but it was in, Ruby, which, for the Heroku CLI way back in the day when it used to be in Ruby, that had its own pains and sufferings, like distributing Ruby was not fun to multiple operating systems and people's computers. Like, Hey, user CLI, but please first start by installing, a language, could be better, but yeah, coming into the Anchor setup. So, some things are smooth, some things not so much. There's some nice Go libraries, but, some things that, they leave you wanting for, I think. So,
Wesley
00:03:19.895
yes, the distribution problem is very solved, thankfully. so check that one off the list.
Darin
00:03:24.654
Was that really one of the first things you wanted to make sure you didn't have to deal with from a CLI perspective?
Wesley
00:03:29.677
yeah, that was, that was high on the list, really. we wanted to make sure that whatever we did was easy to distribute, similar to some of the other stuff we're doing. Like we have, we're also, you know, making it so that you can potentially get your certificates and like, a Ruby gem or, node package or et cetera, like being able to be really cross platform and work for as many people as possible was very, very high on the priority list.
Viktor
00:03:50.255
Darin knows, whenever we saw some new tool, new CLI, where the installation instruction is pip, install, or npm this or that, I freak out a bit. I don't want to look at it anymore after that point. It's just so ridiculous anymore. Today, at least, that we don't get CLIs as a priority. Simple binary distribution.
Wesley
00:04:17.317
Yeah, I mean, you know, if it's pip or something else, it's like, that's great. If that's the only tool you ever have to install, but as soon as you have more than one tool and they'd need different pip versions or whatever else, it just, you know, rapidly escalating dependency nightmare. So no, thank you.
Darin
00:04:33.289
everything we're saying here, it sounds like you wrote everything in assembly, right? That's how you decided to write your CLI.
Wesley
00:04:40.074
Yeah. I mean, I've been around the block a few times, tried different things, you know, and, and yeah, so, moving into Anchor. Go was kind of defined for me, part. That's what the CTO had started working in, so that also was part of it, right? That was how he made the choice. He was more familiar with Go than me. to be totally transparent, part of the pains of developing this in Go was that I had never worked in Go before. So this was my first Go project, and you know, some of it is just wrapping my head around all of that coming from again, uh, mostly Ruby in my background, which, a lot different, approach and, a lot different feel, I guess, to Ruby language than, uh, to Go language. So, just some growing pains there as well.
Wesley
00:05:24.321
I mean, it's, it's hard to beat the distribution and a lot of the biggest pain points we found ways to work through. So that's been very nice. I mean, that being said, like, I do miss. The ease with which you could do sort of more metaprogramming in the Ruby context, right? It was a lot easier to write tooling that helped me to write tooling versus in Go. I feel like I'm stuck in a lot of things where I'm doing a lot more boilerplate or in effect, like copying and pasting a whole thing to another place and making small tweaks because it feels just easier to do that than it would be to try to abstract it, even though it's technically possible. There are trade offs, you know, there's some development cost, even though we get this sort of like distribution runtime benefits. So, that's the thing distributing something that you hope lots of people will be able to use. It's very easy to install and use is that sometimes you just have to eat those pains at the development side of things. Right. So that, people don't have to eat it on the other end.
Darin
00:06:20.445
If people haven't read the blog post yet, one thing that you call out is, okay, you went down the normal suspects and go at least Cobra. Bubble tea, which stack those two together, you get a different use case. But one thing you were talking about is tests. I mean, coming from Ruby, I'm assuming you lived writing tests a lot first, maybe not first, but a lot. And that was one of your big frustrations. It sounded like going down the go route.
Wesley
00:06:51.185
Yeah, I mean, just given my predilections, I guess, the way that I often will approach a problem is to first kind of seek out as much prior art as I can to just kind of get a foundation of like, especially in something like Go where I was a relatively new developer, right? Like probably other people have thought about this problem and have more knowledge and experience to bring to bear. So. Me following the examples of what other people have done is likely to end up with something better than me trying to just invent something that is going to be, basically Ruby poorly translated into Go, right? You see that with the new developers in the language a lot of times where they basically just try to recreate the patterns that they're used to, even though that doesn't make as much sense. yeah, I quickly ran into this problem that I could find tons of examples of BubbleT programs, including, you know, in BubbleT's own repositories and stuff. But almost none of them had any testing coverage whatsoever. And so then I was kind of left with this puzzle of what even is the right way to test this sort of thing. And, you know, where, of course, do we do things like unit tests where we're just kind of testing the logic versus where do we do tests that actually check to see what the final output of this thing is and whether or not that matches up with what we want. And, yeah, it was a struggle to find anything that would give me any kind of traction. the best we found I think ultimately was, BubbleTea within one of their experimental repos had a set of testing things called t test that's based on, the idea of gold files, which is also new to me coming into the Go ecosystem. I don't think it's a Go thing in particular, but it seems more common there, where you have a test that basically writes its output out to a file, and then when you run the test suite in the future, it says, Did the output of this thing match what I recorded previously? If it did, the test passes. If it didn't, it fails. And you can run with a flag basically that says, I actually want to update the test this time around, right? and it's nice because there's a lot of tests, contests for, CLI stuff where, you know, what specifically it looked like mattered to us, right? So recording what the output was and being able to check over time to see if there were regressions, things like that was exactly what we were after. But, unfortunately, nothing could be totally smooth. There were some issues because of the nature of BubbleTea in terms of timing. There were a lot of race conditions that we ran into with their golden file setup where, Just an example, we have a few different things in our CLI where, It's sort of a transient output, I guess you might say, where something is displayed on screen for some amount of time while it's actually happening. So an example is sort of like, I am doing an API request in the background right now. Here is a little spinner. So, you know, something's happening. I haven't just frozen up. Right. But in a lot of cases, once that's finished, we don't necessarily need to continue to have that on the screen. So we'll just clear the screen. Right. Well, BubbleTea test suite is Some runs, the output would display that spinner and sometimes it wouldn't, it just would depend on when it happened to call the methods that would write that to file, And so you'd get a lot of test flake, right? And, as I'm sure you probably have experienced in some cases, having a flaky test feels even worse than having no tests at all. Right? So, definitely things like that helped us to get like a couple of steps closer, but then we felt like we had to kind of like. dig in and figure out how to take that and turn it into what we felt like we really needed. So we created something instead where, is much more predictable. We're still running into some race conditions, but not nearly as many, right? So it's much more common that anything that we say that we want to write to screen, will appear in the golden files, right? so we just have golden files where instead of it being, this is the final output of the thing. It kind of writes, I don't know. It's almost like looking like animation or a flip book or something like that, right? So you see like each kind of frame of what's happening one after another. It's a little bit redundant, right? Because sometimes not that much changes between frames and things like that. But that way we can feel confident that everything was written. not as flaky. we get a lot more consistency. We avoid regression and stuff like that.
Viktor
00:10:43.827
I think you're a very, very brave person. I mean, I use BubbleTea and still use it. And I did not even, I mean, I gave up after 15 minutes. I mean, all the functions that are invoked. Yeah, that's tested, right? But when it comes to Charm in general, especially BubbleTea, I felt like Same challenge as when I was, a long, long time ago, most people don't even know it probably, when I was working with Flash, Adobe Flash. I don't know. Something is going to happen, background, I know what's happening in the background, I know what my functions are doing. What people see is a mystery. I mean, this is me praising you, kind of like, well done.
Wesley
00:11:30.544
Yeah, I mean, it took a few tries and there are some quirky bits in there. They're still, I feel like can be a little bit hard to reason about, but we, Bit by bit said, this is a problem, let's find something that's a good enough solution to it. Okay, that's good enough, let's move on to the next problem that we're having. And, it's working for us so far. I think we've been able to put together a pretty compelling CLI experience. And I think the test coverage makes me feel, you know, reasonably good. I don't know, with something like that where, the things that actually, the Goldenfile tests that are testing the final output, like those are inherently like more integration style tests, right? So I would say we have pretty good coverage there, but it's not that every single possibility of inputs and things that could happen that go through the CLI go through one of those tests because they take a long time to run. They're very complicated to set up, et cetera. Right. But we're able to at least test a lot of the core paths. And that makes it a lot easier to say. Oh, actually, I know that the CLI command that used to work and display things in a certain way doesn't now because of what seemingly is an unrelated change in the API, right? But now, like, this piece of data that used to come across the wire and be available isn't available anymore, and now it broke this, right? So, it's nice also for me because I don't have to be the sort of, like, Sitting around trying to be eagle eyed to catch any change that might be, remotely related to the work that I'm doing in the CLI that one of my colleagues might be doing and prevent that from causing an issue, right? Our CI will actually blow up. And, so that's been very nice. but yes, I felt like we had to just figure it out because yeah, Charm didn't feel like they did us too many favors there. And again, maybe I was totally missing something, but that was kind of my experience. It's similar to what you were saying, right? Of just like. the documentation more or less seemed to say, you want to test this? Like, good luck , Darin: so you brought up in what you just said there, APIs. We set all this up at the beginning, if you read the blog post, that's the first line you should talk about, release new API endpoints. If you're building a CLI, now everything we just talked about, you could have had a CLI that never reached outside of itself, no reason why we can't do that. We did that for decades. It just wasn't cool to call it a CLI then. You just called it a program. But now we have APIs. So in your point, you said release new API endpoints, build CLI actions that need those changes, realize mistakes were made in the API, and you sort of get into a doom loop as you start going further down the list. So when you introduce APIs for your CLI to call, what are y'all doing? Are you doing REST? Are you doing GraphQL? Are you doing Protobuf? Are you doing whatever? Mm I lean towards, rest stuff or rest adjacent. At least I feel like I could get into arguments with people about what it means to be rest, probably. But, definitely more in that vein.
Wesley
00:14:22.928
all right. I don't have as much experience with something like protobuf. and I think GraphQL, I guess, gives me pause in terms of, it's always felt like SQL injection as a service to me, I guess. I, I feel like, um, the main use case that feels valid to me for GraphQL or the most valid, maybe not the only case that's valid is where you're providing. A way for internal developers to quickly iterate and be able to make changes and try stuff and so on, but that you can go over and knock on their desk and say, Hey, knock it off. Or, Hey, make this change. I feel much less comfortable in the context of we're going to release this to the world and just deal with the aftermath. If somebody does something that's very difficult to impossible to optimize and is deeply relying on it, and they're an important customer, we're just going to have to eat that pain somehow. Like that, that doesn't sound very appealing to me. I also believe a lot in, like, trying to provide a good experience to people, and I don't think necessarily telling them, like, you need to figure out our user model and data model well enough to write queries against it in order to effectively do something is providing a very good experience. given those kinds of, edges, my inclination has been towards Rust, where I feel like, A lot of people at least have some amount of familiarity with it. It's relatively straightforward to understand how the pieces fit together. I feel like I can both use it as an interface that I provide to internal customers or external customers and, and feel comfortable and confident in that. You know, I think that we can do optimizations. We can lock things down if we need to, like all of those kinds of things. And so, yeah, that's the kinds of things that I did when I was at Heroku and designed the API there. It's the kinds of things I've done since then. and you know, I've written a lot of client code too in my day, and a lot of that has been around REST clients as well. So, just a space that I feel comfortable in and that I think a lot of people feel relatively comfortable in. That's where I tend to go, these days in terms of like how we actually go about doing it, I'm a big proponent of spec first development, so we use OpenAPI, uh, and generally the way that I would proceed if I wanted to implement a new CLI, command that needed some new stuff like I was working on a command, uh, in the last couple days, uh, wanted to add the ability to create organizations, right? Previously, you could list organizations and stuff, but we didn't let you create it from the CLI. That was all something you had to do on the web, I went into our OpenAPI spec, I opened it up, I added the create action, and then using a tool called, Prism, we can, Spin up a mock server that just looks at the spec basically and can see what an organization should look like more or less. And so if it receives a create organization call, it gives you back a fake organization data blob, And I can actually then proceed from there to, build out that CLI endpoint that I had imagined and see what inevitably I forgot to include in that endpoint or what I didn't get quite right. and quickly iterate by then changing that spec, right? Because. As I'm sure you can imagine, changing a few lines in a YAML file is a lot cheaper and faster than changing, in our case, a Ruby implementation of an API endpoint and the associated tests and everything else that go along with that. And so that way I can often quickly iterate until I see that I have exactly what I needed to provide the interface that I was hoping for, and then At that point, another nice part is then I either can implement to the spec in Ruby or I can even hand it off to a colleague in some cases and parallelize some of that work where we say, we've agreed on the spec now. I'm going to finish building out the CLI that goes with it. Can you finish building out the API that is going to back it on the other end? And then hopefully the two will just meet right up and be happy because, they have a contract that they already agree to. and that's been pretty smooth for us. I mean, we've The not smooth part, of course, was like getting all of those things set up and the tooling related to it and figuring out exactly how How some of those bits and bobs were going to work. but once that was up and running, uh, it's been very smooth. And we do that in, um, CI as well, where, we run a lot of the test suite against mocks because it doesn't really need to manipulate real data. And the tests can run very fast that way. Um, but then we run other things through real stuff where it matters, where we're okay with the extra expense and the slowness that comes along with that. and that's also provided a nice, balance, I think of, correctness versus speed and, and those kinds of things.
Darin
00:18:25.832
When you're making a new call, you were talking about doing the create org. Why didn't you just do that to begin with? You had lists you should be able to create. The reason why I'm asking this, I'm fighting with a vendor's API right now that need to do things with the API. The only way I can do it is through GraphQL. I don't want to deal with GraphQL. I just don't want to deal with it. But yet they have a CLI, but they haven't implemented that thing yet, which is like, we want you to be using this. Go use our GraphQL API. And it's like, I don't want to do that. So how, how can we, all right, I'm going to flip myself on the other side. How can we as providers of these APIs and more importantly, of these CLIs, how can we make sure that we're sort of getting it right as best as we can from the beginning or as a new feature launches, making sure that we, is it really a product problem? We launch a new feature that's available on the API. Shouldn't we also make it available on the CLI at the same time?
Wesley
00:19:33.224
That's a good question. I mean, we're still pretty early days and very small team. Um, and so, you know, we have to make decisions sometimes just based on resources available. So, I think broadly speaking, when I am working on API, I would prefer to, have the whole stack of CRUD operations, you know, create, read, update, delete, you know, whatever. Maybe if there's other operations that are related to an object, like. Build them all out, right. And call it a day. but we right now have few enough people. There's, there's five of us, right. That are trying to do everything that needs to happen at the company. we have to be a little bit more, mercenary than that and a little bit more aggressive, right. Where, you know, if we have a feature that only needs list, like maybe let's just implement list, right. Because, that will get us to having that feature done more quickly. Um, we also have. Maybe the benefit, maybe the curse, I guess, depending on how you look at it of not having, an expectation that our user base is directly hitting the API right now. Like we kind of expect that for the most part, it is just an interface that we are providing to ourselves, right? So that makes it safer and more comfortable to have a limited supply of things available there. and yeah, I think, you know, we also kind of relating to what you were talking about, we're still debating. Sometimes there are a few things where, At least for right now, our thing that is in the web does it better. And so then it's sort of like, do we force it into the CLI, even though it's going to be a worse experience, or do we let it remain in the web context because it's a better experience? concrete example, because I feel like I'm talking like very in the clouds and that is, we debated for a while. If you install our CLI tool and you don't have an account yet, how should you end up? With an account so that you can actually sign in and proceed. Right. And, uh, yeah, we had a debate, like, do we have on the CLI side, something that actually moves you through the account creation stuff and you end up then with an account and you proceed? Or do we have something that basically says, Hey, actually account sign up stuff is really something that's better handled by the browser. So we're going to ask you to go over there and do that and come back when you're done, right? And, I think, Maybe historically was more of a purist, but these days I think, you know, I, I try to do best tool for the job. And so in that context, we said, yeah, let's just kick them over to the browser and say, Hey, like go over to the browser, especially since we, in many cases are doing like an open IT based sign in, right. Where it's sort of like, yes, you can create an email based signup, but you can also just use GitHub to sign you in. Right. And like. Maybe there's some way that we could do that OpenID bounce with the CLI, but it just seems like asking for trouble versus just doing that in the browser, which is what people are used to. You know, I don't need to worry about that. introducing a new security vulnerability because I'm trying to do it in a weird way that people don't expect or, you know, whatever else that might come from doing that somewhere else. so yeah, there's some other things like that. And now given that we've thought about that in other contexts too, like, Hey, this already works in the dashboard. we'd like to expose it in the CLI, but maybe as a first pass, we just have something that says, here's the URL that you should go to, like, we will implement this in the CLI eventually, but we haven't yet. We don't want that to mean that you can't. So there's a lot of different ways that you can get there from here, that you have to go and navigate through the dashboard and find the right place to actually do this. Like we can help you out in terms of pointing you to the right place, but this is just like where it is now. So to your point, I think I try to make sure that there is a way for someone to do something, but it may not always be the same way, if that makes sense. we also, even within the CLI, have some things where. like one of the biggest commands is this command that's just called LCL. It's to help you set up local, certificate management stuff so that when you're running a server locally, it has real, signed certificates that your system will recognize, you know, and it actually has a bunch of sub steps that it does, and it's possible that you can run each of those steps individually, and sometimes you would need or want to do that, but you can run it as this big amalgamated sort of like workflow as well. and that's another example of sort of like. I could have just released the thing that had this whole series of commands and the way that you would run it would be to go to a webpage that was a tutorial that said here are the 10 commands you need to run in order. I've certainly seen that from a lot of other CLIs, I'm sure you guys have as well. But we said, ultimately, like, we, we don't want that to be the experience that someone has. We want them to be able to come in and run one command, especially if we know what the answers are, right? We know what the 10 commands are. Like, we don't need to tell you to go reference a document that we wrote somewhere else that has the 10 commands. We can just have something that runs the 10 commands in order and, and ties them together on your behalf. so that's been another big change that I think, again, is like a little bit different than I've seen other CLIs. again, I guess going back to sign in, one other little thing that's kind of weird. You're probably used to this in the browser of like, if you try to run a command where you're unauthenticated, that requires authentication, you know, what would you expect? It's going to kick you over to the sign in page. You're going to sign in. It'll kick you back to where you just were, you proceed on your way. Right? I at least don't think I've ever seen that in the CLI before, but that's exactly what we did as well. Like if you try to run a command, it requires authentication. You're not authenticated. You're unauthenticated. We know that. We know how to authenticate you. Like, why do I have to just, like, crash out the command and say, like, You did it wrong. Please do it right next time. Right? basically the experience that I've become accustomed to in CLI, but it's not a very pleasant one, right? So instead we say, Oh, hey, we noticed that you're not signed in. Let me help you with that, We get you signed in and we say, great, you're signed in now. So that part is checked off. We know that the next step you wanted to do was probably the one that you were trying to do before we realized you weren't signed in. So let's just continue on our way. Right? so again, it's like, I guess not trying to be too stuck in exactly how things have happened historically or how we've seen other tools do it or how even the CLI has done it versus the browser. We're trying to borrow from a lot of different contexts to just provide the best experience possible.
Viktor
00:25:14.332
So you're treating essentially CLI as the first class citizen instead of a helper in a way, right? What you're describing is something that I would be surprised if I got to say, let's say browser based service that can, that of course there are cases where you don't have to be logged in, but I'm now talking about browser based service that you cannot access without being logged in. It redirects you to a login page. But then the CLI of that same service doesn't, right? It kind of feels like it's, it's not the, it's not the first class citizen in a way, right? But in your case, it's the other way around, if I understood right.
Wesley
00:25:55.997
Yeah, I mean, I think there are some cases too where like there really are like features and things that the CLI does that the browser doesn't do at all, can't do really, you know, especially as we talk about, again, things related to certificate management locally on your machine. Like we need to be able to do things on your machine, right? And the browser isn't going to be able to. Update your trust store to include the certificates that we just mented for you, right? Like that needs to happen somehow, somewhere. So, yeah, we really lean heavily on the CLI to provide that tooling. I mean, it's also like, as you guys were saying, you spend so much time in CLIs. Like we want it to be something that people are comfortable with. above and beyond even what their experience with other CLIs are, just like we do that with sign in, you know, it's the same thing. Run a command and we know you need to select a service and you didn't provide one with a flag. Again, instead of crashing, we say, Oh, like we know how to look up what services you have. We'll just provide you with a selection list and you can choose which one you want to proceed with. Right. Because at least in the interactive case, like why should you need to know that upfront off the top of your head? if we can just get it for you, right? Like we do also allow a flag and if you provide it, then we don't put the selection up. But like, again, like. We don't want it to be that like, you have to get all of your ducks in a row and run the command exactly right up front. If it's something that we can help you to resolve, then we will help you to resolve it so that again, better experience, smoother, faster. And, you know, I think the other thing that's a little bit unique about the use case of certificate related stuff in particular is like, If we really succeed, you actually are going to run this like one or two times and walk away, and you know, it might be months or years before you run it again, right? because, hopefully if everything's set up correctly, we use Acme and stuff to do the actual distribution of the certificate. So like, they're just going to keep renewing themselves and stuff. You're not going to need to come back to this that often. So I don't even I don't want you to necessarily build familiarity and comfort with the CLI tool. I want it to do what you need it to do and get out of your way, I don't expect that you're going to like memorize all of the commands and all of the flags or whatever because you're in there every day using it. If you are, then like Something has gone awry, probably. Like, that's just not what we expect in the cert case. I guess I'll wish that more CLIs were like that. Like, that they didn't expect that, like, I don't know. They were first and foremost in my mind, and I was going to, you know, memorize their man page because that's the way to use the tool, right? Like, for some things, sure. Like, I use Git constantly in my development, right? So I know lots of Git. Not all of it. I mean, parts of it still are just, like, totally befuddling to me. I was running into some of that today. But.
Wesley
00:28:26.833
I mean, for the core stuff, you know, there's like six or eight commands that I use all the time. I know those well, I'm pretty comfortable with those. But you know, today I was doing like a rebase or something and they're like, this was changed by us or this was changed by them or something. And us versus them means a different thing in the rebase context than it does in any other context I could imagine. I had to like look it up and I was still confused. And you know, like, Very complex tool. So, easy to, uh, I guess, poke fun at in terms of how well you could possibly know it. But I don't know that it does you very many favors in terms of helping you to learn it, or helping you to get better at it, or helping you to do things. Like it's left pretty much like, this is an exercise to the reader to figure this out. It's a very complicated context and system. Like, good luck,
Viktor
00:29:07.504
Going back to what you said earlier, correct me if I'm wrong, you said something along the lines that, hey, in an ideal situation, users of your CLI use it once every two weeks, you're just simply there, it's doing what it's supposed to be doing. Does that affect business in some sort of way? Because, you know, in my head, from the business perspective, it would be easier to sell something that, that I see every single day, potentially all the time, right? Uh, because, you know, you interact with it all the time and so on and so forth, and you have higher perceived value of it. Am I completely off there?
Wesley
00:29:49.654
no, I don't think so. I mean, I think there's a reason that people talk about in terms of business metrics and stuff of like, you know. Um, I don't know how daily active users you have or monthly active users, you know, like the degree to which there is engagement with the tool. Right. I think what's tricky with what we're working on and, you know, like we're still working this out, we're still young, we're still building our customer base. Hopefully it's growing, you know, steadily, uh, could be faster. That would be great. But, Like I said, with certs, you know, there, there is a long tail of how much it is, changing or not. I think, you know, it's like anything that is oriented around automation, if your product is in the automation space, like ideally you set up that automation and you don't have to poke and prod the automation constantly. Or else, like, what's the point of even having automated it, right? You're, you're in for extra trouble now because the automation is a layer of abstraction that's not providing value to you, right? So for us, again, with Acme, it's, it's similar to like Let's Encrypt and stuff too, right? issue is like, it's very high value and it's very important, but infrequently, right? So it's, it's tricky again, like you said, I think getting people to understand that value proposition, like. Is hard. You only have to have like one time where a cert expires when you didn't expect it and weren't ready for it before you understand very deeply the value of that, I think, or have to deal with a problem even of just like distributing certificates to new and different places, Again, very painful, but once you've done it, oftentimes you don't have to deal with it again for a while. I mean, you also run into the problem. Like if you, you know, whether you're using our tool or not, like you don't have to do this thing very often, but you do have this context where, especially historically, the thing that you did have to do. I'll say it infrequently, was impossible to remember and inscrutable, you know, it's like, what is the open SSL invocation that I needed for this again? Like I did it six months ago and I know it worked. It's worked since then. Now I need to do it again. And I do not remember what it is. It's like very arcane, you know, it's like I have to cast this magic spell almost of like putting this particular set of parameters together in order to get the working certificate. And then I have to remember how I got it to all these different servers and how I then Made them understand it. And I need to do that in a way that it's sort of not having downtime in the middle as that transition happens. And, you know, it's like, I don't know. I don't want to have to remember all that stuff. Right. I would much rather something just do it for me. but it is tricky, right? It's similar, I think, to. What I feel like I always ran into when I was working on things at, Heroku, and I tried to explain that to people who had a lot of cloud, expertise and knowledge. They'd done all of these things, themselves. They would be like, oh yeah, I've had to do that myself. I totally understand how painful that is. I would much rather pay you to do it. But when you talk to people that were relatively new to it that didn't have those scars. They would be like, Oh yeah, like we'll just write our own Heroku internally. How hard could it be? And, you know, you can't help, but almost laugh of just like, Oh boy. Like if you're asking that question, like I guarantee much harder than you think.
Viktor
00:32:38.916
I feel that those, if you can wait, those are the best customers you can ever get.
Viktor
00:32:45.976
that, and I'm paraphrasing what you said, those that say, oh, I can do that myself. I can, I can, I can create my own. Why would I pay Heroku? I can do it myself. You just need to have patience to wait for a year or two until that sinks in, in a way.
Wesley
00:33:02.641
Yeah. I mean, it's like, you know, the person that decides that they're just going to repair their own car or something. Right. When they come then to the mechanic two weeks later and the car won't even start. The mechanic now has a, probably a pretty good customer for a while as they get that thing back up in order, you know? So it's tricky, I think, to sell on something, at least in some cases that really relates to a pain. So, I mean, that's one thing that we're really trying to, I think, Figure out is like, we don't want it to be that the only reason you buy this product is because you've had that pain and you don't want to have that pain anymore. We also want to help people understand like the value of doing it this way. Right? Like, The value of doing a true secure context locally instead of the sort of pseudo secure context you use in localhost, right? Like, you know, you can like actually do local testing and development against payment gateways if you have a true secure context and you can't if you don't have that, right? And you can use a lot of the, exciting new, browser features that, touch things on your system, like your webcam and other things that aren't allowed outside of a true secure context, you know, like. All of that is cool, and it's valuable, and it's like, again, like, uh, it's sort of a vitamins versus painkillers thing, I guess, right? Like, yes, I would love to take your pain away, but I would love even more to give you a new capability, something that you can be excited about, something that makes you stronger and better moving forward.
Viktor
00:34:16.628
Oh, now you brought up a wrong example. I can bet in anything that painkillers sell much better than vitamins.
Wesley
00:34:25.288
Yeah. You've had the pain, so, I mean, you're wanting to get rid of it. Not everyone is there, though,
Darin
00:34:30.660
You've already stated that Git CLI is your favorite CLI, not. I had this thought and somebody was talking about this the other day and it just sort of hit me. The Git CLI is an expert tool for an expert user
Darin
00:34:54.388
okay, fair enough. But isn't that where most CLI authors start? Let me rephrase that. Isn't that where most software authors start? We're writing software and we expect everybody to be an expert from day one. I
Wesley
00:35:09.374
mean, I think we often do, but I don't know that we ought to. I mean, that, that, that's maybe a separate question.
Darin
00:35:14.594
well, no, I, I, you're, you're right. We shouldn't do that. We should go back to the beginner. It's like, nobody's ever seen this before. How are they going to know what to do? So going back to CLI, I came up with the phrase, you are basically dynamically completing for me. Thank you. What I didn't know that I was needing completion for. And all without AI, I would assume.
Darin
00:35:40.933
Okay, good. Okay, good. No, just, you should have stopped at no AI, period. You don't need presently thrown in there.
Darin
00:35:52.163
I, I, I don't want AI getting into my certificates. We'll get to the certificates again in a minute.
Darin
00:35:58.783
Okay. We're joking about Git. What CLIs do you think are doing things good right now in general?
Wesley
00:36:04.732
Yeah, uh, it's a good question. I mean, one of the ones that I often go back to is actually the, um, GH, which is Git adjacent. It's the GitHub's CLI, right? there are parts of that sometimes that I wish were a little bit different, but overall they do a really nice job. it's also based in BubbleTea. So in terms of when I was looking for examples, that was one that I did look to. They do have very nice test coverage. There is a lot of like, harnessing and infrastructure that's very specific to them and how they do their tests though, which made it not something that I felt like I could easily replicate, without a lot of, troubles, I guess. Uh, it's not easy to just replicate that, but, but yeah, GH overall, super nice. as we said, Git leaves something to be desired. I think a lot of Linux tools are very challenging in that way. I think, like you said, I think, That categorization of like expert tools for expert users, I think definitely fits the design aesthetic, I guess, that I see around. one thing that maybe sets apart how I think about a lot of this sort of philosophically or whatever, I guess, is that I really see when I'm designing an interface for end users, the goal is. Maybe more about knowledge transfer than it is about them getting something done necessarily. I mean, I want them to get something done, obviously, but I want them to be able to learn along the way. So we also have a lot of like output and stuff that happens in our CLI that is to help you learn how certificates work and what it is that we're doing and why it matters, even though we could hide all of that behind the scenes. We could just say. splines, you know, spinner until it was done and then say, you're good to go. but we want to take that opportunity to help you to understand the underlying mental model, the underlying data model. because eventually, at least some of our customers will get to a place where they want to do things kind of like we were talking about before, where. They want to do something that is new that isn't quite in the CLI yet, Or something that's at the edge. They have a more complicated use case that is specific to them that we don't support, but that they could support if they just knew how to turn the right knobs and pull the right levers, so the more that we can educate them and empower them to, interact with certs in new and better in different ways, the more, they can use our stuff or just encryption in general to be more secure, to be better developers.
Darin
00:38:20.258
Should every piece of data on the wire go through a cert, be encrypted, whatever you want to call it at this point?
Wesley
00:38:27.971
Probably. I mean, we've seen that movement in terms of, browsers and stuff, right? Like browsers are moving more towards almost everything on the web is HTTPS, Thank goodness for Let's Encrypt and other things that made the server side of that equation easier. But, it doesn't totally solve it. You still run into some issues, like, Internal certificates are still kind of a pain, especially if they aren't exposed to the web in some way, where they can talk to Let's Encrypt, right? Like, oh, if you have a homelab or something, you know, like actually getting certs set up on your homelab can be a pain. But in terms of doing it everywhere, yeah, I think it's like a defense in depth thing. The example I sometimes give is like, just because your house has Fire extinguishers doesn't mean that you don't also want smoke alarms. Like you, you want to have many different layers of things, right? Or, you know, if you have a burglar alarm that doesn't, and you should leave your doors unlocked, right? Like, you don't want to just rely on one thing to secure things. So even if you are running your servers inside of a VPN or something, like if somebody gets inside that VPN, you don't want them to just like, Keys to the kingdom. They can do whatever they want. Sniff whatever they want. You would never know, It's much better to be able to say, yeah, that is one layer is the VPN. But if they get in, they still would need to be able to get a hold of the certs or do things with the certs. And like, maybe they still could depending on how they had infiltrated. But, you know, extra layers means, you know, I think in many ways, the barrier wasn't that people didn't know that that was the right thing to do, or that it wasn't the right thing to do, it was that it was too much of a pain to do, right? And so trying to lower that barrier, make it easier, simpler for people to just do the right thing, I'm hoping will drive more and more people to do that.
Darin
00:39:58.681
If you had to start all over again with the CLI, you said it had already sort of been started by the CTO. What would you have done different? Like if you, if you were being brought in day one, it's like, Hey, we've started roughing in the API. We know it's not even completely usable, but that's not Greenfield, but the actual CLI is Greenfield. How would you approach that?
Wesley
00:40:22.236
Uh, I don't know that it would be too different from what we ended up doing. I mean, it's always hard because we learned a lot along the way, right? So I'm not the same. crossing the same river, but what is it? I don't know. You can't cross the same river twice or whatever, because either the rivers change or you have that kind of thing. Right. Like we learned, quite a lot. and you know, a lot of these things that I was talking about in terms of, having the sign in just happen or having the selectors that just work or whatever, like that wasn't something that I came into this thinking of necessarily. there were some things that were kind of adjacent to that I'd thought about, but, yeah, I think. A lot of it just took us iterating through and trying things. I think I would have done some of those things sooner probably. but some of them also involved just me like having arguments with the other people in the company and convincing them of trying this because as I think you guys realize, this isn't exactly like other CLIs, right? Like people, at least in my experience, don't necessarily talk about and implement them in quite this way. I, I think they could or should, that would be great. I think that they would provide better experiences. convincing people to give it a try was a little bit of a challenge at first. I'm glad that coming into this, I had known the founders for about 15 years already. They were willing to give me, I think a little bit more slack and a little bit more opportunity to just run with something and try it and show them demonstrate the value. whereas, you know, if I was brand new and they didn't know me from, anything, that might've been much more challenging.
Darin
00:41:38.589
so if you're happy with sort of where you're at, you probably wouldn't change anything major. What would you tell people to do? Like if they're now being tasked with, okay, Hey, we need an API and we need a CLI, how how would you tell them to start, especially if they're the freshers, right? They're the people right out of school or whatever. And it's like, Hey, these guys are cheap. Let's just go ahead and put them on this. Cause how hard could it be?
Wesley
00:42:05.863
mean, there's a couple of different aspects to it. One is just, similar to other things, although I haven't often heard it in this context, I guess. one of the things that I think makes you a much better producer of something is to become a connoisseur of it. And so if you want to make good CLIs, you should go and check out some other CLIs and see what they do and don't do. Develop a taste for what you like, what you don't like, what frustrates you, what excites you, you know, like, pull up GH. I think all of us kind of are nodding in agreement that they seem to do a pretty good job there, right? Like, go through that and try to get a feel, a little bit of like, why do you like it, right? Develop an opinion about what the parts are that are good about it, not just, uh, you I know that it feels good when I use this tool compared to other ones, right? But why is that? What are the parts that stick out that make it smoother or less smooth? similarly, like, in my background, I wrote a bunch of clients for a bunch of different, IAS, cloud providers, right? And so when it came time for me to write APIs, which is something that is, you know, Broadly speaking, slow, costly, you don't, you sort of don't get that many swings at it in your career. I had a lot that I could draw from, right? I didn't have to totally start from scratch because I could say, Hey, I know from my experience of sort of like being a critic in a sense of other APIs and being a connoisseur of them. That this is a pattern that I like, and I know that this is a pattern that I don't like, and it became even more handy when it came past the point that I was just doing all the work on an API to there was a team of people that were working on the API and they would come to me and say, how about we change it in this way, And then I, it wasn't just that like my intuition was driving everything, I had to go through the struggle of turning my intuition into an explanation to that person as to why that was a bad idea, right? Like I could say, that will taste bad, right? Like, in terms of the, stretching the connoisseur metaphor a little bit, right? Like, I have tasted a thing like that and it was not delicious, right? So let's not provide not delicious things to our customers, right? so that's a big one. Uh, the other part is, I guess also relating to not getting to take that many swings and it being very costly is as much as possible, I try to do iteration on whether it's APIs or CLIs or whatever else in sort of the cheapest way possible before moving on to progressively more expensive things. And so I mentioned a little bit of this earlier. So. When we're working on APIs, we write a schema that defines what the API ought to be like. We run mocks of that to do the CLI development against, right? Those both are ways to make it cheaper. On the CLI side, more and more of what we've done, even in the early days, what we did was I would often write a simple program that would just spit out to screen more or less what we thought it ought to spit out to screen, that behind the scenes was not doing anything, All it was was like fancy way to print a bunch of lines to screen, but it was able to give us an opportunity to actually see what it would look like or what it would feel like to do that. Um, now all of us have become comfortable enough with the golden files that we were using before that often I'll just like make a fake golden file basically that displays what the output will be because that's even cheaper and easier than a command that does a bunch of print lines, right? because it's just a text file that I'm editing. but then that way I can quickly iterate because with a lot of these interfaces, in my experience at least, and maybe someday this won't be true, like I always get it wrong the first two, three, four times or something, right? Like it's not usually that. Whatever the first thing that comes to mind for me is, is the thing that I'm satisfied with. I usually have to iterate a few times to figure out, Oh, I forgot that thing. Oh, right. I need to include this thing that I didn't, have in mind. and so again, like doing that iteration in the really cheap place, not the Let's implement this whole thing from, soup to nuts, end to end. Let's do all of the CRUD operations for this resource or whatever, and then realize that you made some fundamental mistake in your understanding of the problem, and have to go back and re implement all of that. that's the other big thing. So, move in the cheap places, learn from other people, right? Like, I think those things can really, uh, get you a really long way.
Wesley
00:45:53.228
yeah, it's, I don't know. I mean, mostly I'm, I don't know, some of the things I described, I guess, in terms of what we do are kind of like on the edge of some of that, I guess, like, you know, having the multi selects and stuff like that, right? Starts to be almost Tui like. I still like it to be mostly like, I've asked you to do a thing. Maybe you're a little bit interactive in the process of doing that thing, but then we're done, you know, like I finished this thing and we're out. It's not, I'm entering into your environment and then doing a series of things or whatever, I mostly like it to stay in that more functional, space, I guess. That being said, like I'm a VEM user, right? VEM is a very 2e ish thing, right? And that works well for me for what it does, but I don't want that for everything. Right? Like, I don't think I want a 2e for Git. I'm fine that Git is a functional tool that I, or GH or whatever, right? That I say, do this thing for me and it does it. And then I'm still back at my terminal going on to the next thing. so there are times and places, but I mean, I think it also is like, like I was talking about, like we are still having the CLI just kick you to the browser sometimes, right? I think there's a similar thing there of like, I don't know that the CLI necessarily, or Atui for that matter, does it better. It can do it equivalently, but if it's just going to be equivalent, like, is there really that much more value in it being in that context versus some other context that already exists and is probably going to be cheaper and faster and more familiar for people? my sense is probably not like, you know, if it's going to be an email client or whatever, like my email client and my browser is fine, I don't find myself wanting that much. Like, I don't feel like I need a 2e version of that. but to each their own, I guess.
Darin
00:47:30.047
No pun intended. TUI is just, um, bad dad jokes. Sorry. So Anchor can be found at anchor. dev. You've been talking around it the whole time. What does Anchor do?
Wesley
00:47:43.106
Yeah, we're, trying to make encryption effortless for developers. because like I said, I, I think we realized that it's the right thing to do, but we've been bitten by just how much of a pain it is to do, right? Self signed certs, huge pain. and there's not a lot of other good options for local development in particular. Um, there are some things like make certs. pretty good but there's still a good portion of that experience that's sort of left as an exercise to the user and so we're trying to really take that and make it as streamlined as possible as you can imagine probably from some of our discussion about various things where we know hey like we actually know what you would need to do here so let's just help you with the next step that Is a lot of the experience really, right? So the idea is really, you can go, you can install the tool, and you can just run anchor LCL and it will do what is necessary to get you set up where your app has, integration with, Acme and uses Acme then to pull down a certificate when you boot it up and can renew that certificate automatically for you. And just. Install that certificate into your local trust store, so when you boot up your browser, you actually have a true, trusted, context, and you can do whatever it is you want to do, again, like, trying to just make that a natural thing that people will do, because I also have the experience from my career of, looking at an open source project or something, and somewhere buried in the readme is the Bye. Bye. By the way, here's how you turn off TLS. You should never do this, but we get asked so much about it because it's so much of a pain to properly configure this, that here is the option that you just set and TLS will be turned off, right? like to put those days behind us.
Darin
00:49:13.774
So again, Anchor can be found at anchor. dev. And we'll have all of Wesley's contact information down in the episode description, along with a link to the blog post that we were talking about at the beginning. Wesley, thanks for being with us today.
Wesley
00:49:26.540
Thanks so much for having me. We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox. com contact. If you subscribe through Apple podcast, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox. com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.