00:00:00.000
This episode is sponsored by Allstacks
Jeff
00:00:03.005
Are there more deployments going through? Yes. Are there more tickets being closed? Yes. Are there more confidence in the ability to take an idea and vibe it through to come up with something? Absolutely. Are we truly more efficient? Do we have more throughput? or as the Dora report talked about, creating more instability? Also, yes. There's problems that are happening. We're not really seeing the throughput. We're not really seeing the increase in quality. In fact, while individuals report better individual productivity as a team, we can't really answer the question of are we shipping more value? This is DevOps Paradox, episode number 368, The AI Productivity Paradox
Darin
00:01:44.330
Viktor, you use AI for daily development. I use AI for daily development. A lot of developers are using it. 90% are using it. Most of them are saying it's helping them be productive. Is it making you more productive, Viktor?
Viktor
00:01:55.190
Let's put it this way. If you tell me no more agents for you, I'm uh, growing tomatoes. I'm not going back to software industry. That's executive decision from my part
Darin
00:02:05.820
One study found that developers who felt 20% faster were actually measurably slower. So here's the question for today. What if a whole team is confidently shipping nothing faster? On today's show we, we have Jeff Kise from Allstacks. Jeff, how are you doing?
Jeff
00:02:30.180
No, Kais. we have to make it messy. If, if you speak more Latin languages, people say Kais 'cause it looks like Reyes. and in English most people say Keys, but it, it's Kais
Darin
00:02:40.914
and I even have it phonetically in front of me and I s- did it with a dash, so it's KAIS. Did our story sound even familiar there of what's happening? In fact, let me ask it a little bit differently. We're not debating whether or not AI is helping type code, right? We can see it typing code really fast. What we're really trying to figure out, is it helping us ship any faster? What do you think, Jeff?
Jeff
00:03:04.684
For sure. Are there more deployments going through? Yes. Are there more tickets being closed? Yes. Are there more confidence in the ability to take an idea and vibe it through to come up with something? Absolutely. Are we truly more efficient? Do we have more throughput? or as the Dora report talked about, creating more instability? Also, yes. There's problems that are happening. We're not really seeing the throughput. We're not really seeing the increase in quality. In fact, while individuals report better individual productivity as a team, we can't really answer the question of are we shipping more value? there's some reasons why that I believe are part of that story. But is AI helping developer productivity? 100%.
Viktor
00:03:54.448
everybody says that on individual level AI is helping whatever that means, the logical conclusion must be that then there is something structurally problematic on a company level, a process level, right? Because if I'm more productive and you're more productive and Darin is more productive, but when we combine our productivities, that does not turn out to be more productive as a whole, then my only explanation can be, okay, your process does not work for this
Jeff
00:04:28.566
Yes. I like the explanation that AI is an amplifier, and it will amplify the good and the bad. And I think the challenge of the bad is so much that it's offsetting some of the good. if I remember right looking back at some of AI's impact according to the research that was done last year, individual effectiveness like went way up, but software delivery instability also went up. Amazing thing that happens to pipelines that were mostly right, when they're mostly right and you put more through them, you got more failures and more issues to go solve, which means higher incident rates, higher deploy failures, and the rest of it. The most notable example from my point of view was a recent session we ran to exemplify a new feature. We, we ran a proof of concept, and part of that proof of concept included this text editor as part of it. And w- not thinking it through, we just built it all, and the agents went to work and did great things, including building a text editor. it's happy to do whatever we feed it and whatever it sees and wherever it goes. Is that exemplifying what problems we see in the industry? I think so. I think there's a lot of do platform components get used well? Are existing APIs being duplicated? Are bad patterns being replicated because that's what we did in the past? Maybe. How much are we learning from agentic development from what we've already done? answering that question, I think is at the heart of where AI is amplifying some of those negatives.
Viktor
00:06:16.747
But apart from amplifying negatives, which I-- I mean amplifying everything, both negatives and positives, I feel that the problem might be that people driving AI and agents might not feel comfortable in the new role, right? Because you have a typical developer in a stereotype enterprise that, hey somebody made decisions, somebody specified exactly what you should be doing, and so on and so forth, and you just need to do as you're told. And now we're in a situation where that same person is supposed to be telling others, and by others I mean agents, what to do. And that person never did that, right? Kind of like, "I know how to follow the rules. I don't know how to come up with rules. Not my job."
Jeff
00:07:08.774
I gotta tell you I, I see this a lot. Part of what I do in my job is I talk to a lot of different companies and, customers, and walking them through… Or not walking them through, but I'm, part of their journey as they rediscover their new processes and what are these new roles. Developers now see themselves in part as designers, 'cause you can. You can just jump out to tools and go do it. Developers see themselves in part as product people, 'cause you can. Product people see themselves in part as developers. So the roles are very much blending. The people that were doing definition all along, starting all the way from top-down of, "We're gonna have this initiative, and it's gonna drive this kind of value," all the way into, "Did it work?" is being reimagined. The people that thought they knew so much now need to work differently. is doing a whole disruption of how the team operates, the size of the team, the roles, the handoffs, the measurement of value through. And that's where the conversation of the impact of AI to the software industry is super fun. we sit at the precipice of what will be an entirely new model as big or bigger than Agile was back in the day, or even Waterfall. The whole methodology of how we build from beginning to end is different. And now that the, the genie is out of the bottle, it, it never will get put back in. This is a new world we live in.
Darin
00:08:43.395
I'm thinking about what you were just saying there. You're talking about value and flow. I have a feeling you come from a bit of a value stream background. Uh, Y- 'cause that just rolls off your tongue
Jeff
00:08:54.089
Yeah, it does. I-- One of the things that frustrated me a long time ago was looking at people that talked about wanting to go faster. so we would sit in these cycles of analyzing software delivery, and they would really optimize how fast can we get that deployment, how fast can we get from even an idea into developing something when there were six to 18 months on the front end of that while they're figuring out which kind of initiative should we build. They would get a, feature request, and it would sit for a year waiting to decide, is that the right one? Instead of just building it. been in other environments where even after it was coded, still now has to get into some really long release cycle where it takes another quarter or six months to actually get in the hands of customers. Meanwhile, where the focus was let's speed up development because we have all these metrics about where they're at and what they're doing, each individual, how many keystrokes are they pressing, how many lines of code. Please, I hope we never end up back in that world again, 'cause no one cares how many lines of code. Let's look at value. Part of that discussion led me to part of the journey I've been on in my career. I worked at Planview. I was at Plutora. I helped start the Value Stream Management Consortium. We managed delivery looking at the entirety of the life cycle of software from idea all the way into delivery itself borrowing from lean manufacturing, the basic concepts of flow. How long does it take to get something done? I don't just mean coded. I mean from the time you thought about it into production and actually getting value. What's your typical time to get a work item completed? What is the load that has on the team, meaning how many different kinds of work items are there at a particular time? All that led into what became the Value Stream Management Consortium. That has been reformed into a new organization called Flowtopia. It is out there still in the industry, and highly suggest get involved
Darin
00:11:04.263
So here's a question for that. When the AI coding tools showed up, did it change the flow conversation at all, or did these tools just make the old bottlenecks that much more obvious?
Jeff
00:11:17.913
Y- yes to both, actually. Did it change? 100% it changed. It changed that you could take an idea, if you knew really what you were going to build, and get it through to production so much faster than you've ever been before, and look at the consumption, the value realization side of that equation faster than ever before. Does it happen that way? No. Some of the old bottlenecks are still there. Back to your second point of did the bottleneck move? Yes. A bigger part of conversation, what I'm really passionate about today is about where that bottleneck moved to. the new world of what needs to happen next in this transformation is not an engineering problem. It's a product problem. It's a definition problem. It's an organization and team topology problem. It's a how do we act and behave differently so that we can keep up with the tooling that enables us to build things on a whim and decide if we built the right thing or not. What does that look like and how do we do it in a way that is sustainable at an enterprise?
Darin
00:12:27.699
Well, Even in enterprise, does that matter? Even if it's a small business, it should be the same thing. I should be able to get something shipped faster now, or h- I have an idea this morning, I should be able to, to be testing it out by lunchtime to figure out if I want to ship it at dinnertime
Jeff
00:12:42.753
You're 100% right. I throw the word in at an enterprise, I mean at scale. easier for a small company that's not ladened in technical debt and decades of code to try to navigate and correlate across teams. Harder for these teams that have to do this at scale, meaning the number of people, the number of technologies, the number of historical components and things that all have to work together that have a certain amount of governance and controls that must also be in place. That's where this gets super interesting.
Viktor
00:13:16.464
I- I'm glad that you mentioned technical debt, because one thing I don't fully understand is until recently people were saying, "Hey, we know that we have stuff that we don't want. We know that we have mainframe, we shouldn't have mainframe. We know that we have this and that, whatever, we shouldn't." And now that it's extremely cheap and, and we, we are not fixing all those things because it's too expensive. We don't have time, we don't have bandwidth, we don't have money, we don't have the things needed to fix it. Now that it's extremely cheap to do those things, why do we still have them? Why do we have technical debt? As in not technical debt generated by AI, just to be clear, technical debt generated by our past decisions or decisions that could have been good 20 years ago, but they're not good anymore
Jeff
00:14:07.034
Which every decision we make today is probably the same boat of it's gonna be a new world in 20 years. I posed that same question to a head of product at a healthcare company and talking about their AI transformation. His response was really interesting. Talked about that their AI journey, they decided explicitly while they had pockets that were dev focusing on how to implement the technology, they just waved their hand and decided that the development side was gonna get solved, and that the cost of coding would go and be much less. Where they saw the deficiency, including in these legacy platforms, is their ability to discover and manage the original requirements of these systems so that they could rework them and bring them forward. The idea in reworking them, they were left with this choice of do they go out and try to discover all the original requirements by sniffing through the code, looking through behavioral patterns, looking through the data? Or do they bring it forward, and decide what things do they keep or roll together or so forth? They decided on the latter, and in the journey they learned a lot. In deciding if the code was cheap, that meant what their real work was to discover how do these systems operate, it wasn't just that the people were, left the company, like in some cases, like the people were no longer on the planet, and they needed to discover things that they weren't gonna discover any other way. So they had to reverse engineer all this to then bring it forward. The point is, is every company at scale faces this same problem. What do you do with all this legacy stuff, and how do you modernize it? And how do you make progress? Because that debt, if you will, is a boat anchor that slows down any future innovation that you do. You have to service that debt, modernize. There's gotta be a certain amount. Back again to the, value stream side. One of the key metrics looking at value stream management is, what we call flow distribution. The idea is that there is a certain amount of work that you do in any given cycle that will be a breakdown between innovation, debt, risk, and Oh gosh, I'm trying to remember this. It'll be a certain amount that is debt, risk, innovation, and defects. And the combination of all these things have to be evaluated in combination to decide where you're really gonna place your, bets. Y- whatever you're using to generate the code is great, but you need to take into account of the combination of all that so that you're meeting your goals. Why is this relevant? If you use AI and you generate only features, will there be bugs? Of course there will be bugs. If you use AI and you only address defects, will those defects continue? Yes, because you're not reducing your debt. At some point, as you look at the overall stability of systems, you can find an ideal profile of how you should be making that investment. Who should be making that investment? Product. So flow distribution sits at a pinnacle point a, critical measuring stick for how you look at everything you go build. A certain amount of work you should do should be to address security and vulnerabilities. A certain amount you should do addresses risk, and incorporate that with innovation and defects
Viktor
00:17:37.034
I agree with everything you just said, except that one of the main reasons-- So we have depth, we have issues, we have bugs, we have all those things, without a doubt, and we will never be free of them fully. But the main reason wh- why we have them in quantities we have them is because somebody said, "Okay, we are 1,000 people, and we need 10,000 people if we want to put this to a manageable level." And then somebody else answered and said, "You're not getting 10X increase in headcount. Forget about it. Deal with it." And then you need to make those choices. But now that I can effectively increase equivalent of a headcount very cheaply, I consider AI actually to be very cheap compared to human labor why would somebody have more than a manageable number of open issues?
Viktor
00:18:36.935
I, I'm, I'm not going to say all intentionally, but put them to a manageable level. Let's put it that way, right?
Jeff
00:18:44.445
Such a great question, right? Have we discovered all of our defects? Do we know what the impact is of every time we fix a defect? Is there a large defect because it's technical debt? Is it a design technical debt or architectural technical debt? Is it a platform technical debt? There's so many different ways that this comes out that has to be addressed. Back in the core DORA metrics themselves, which I love what they did the foundational piece is that what they discovered was basically with increased AI usage, there's an increase in platform instability for all the reasons we just talked about. I had a friend that bought a Tesla, thought it was super cool. He told it to park itself, and it promptly parked itself and crashed in the garage, dented up the bumper. It was awesome. We just feel so bad for the guy. D- how much do you really trust AI 100% to say, "Oh yeah, 100% of the time it's gonna be just fine fixing this bug and deploying out to production"? If your failure rate is such that it is low enough that you can trust it, great, but most pipelines are not there. So even just fixing a bug and changing anything still means you've gotta run through that pipeline and see how well that all works
Viktor
00:20:13.231
I don't trust humans 100%, just to be completely clear and transparent. So I don't trust people, I don't trust agents, I don't trust anybody fully, Nobody gets a blank check. But me not trusting agents 100% does not mean that it cannot do 20%, 80%, whatever the percent is, and I'm still winning, I'm, I'm going to be extremely conservative here and say, "Hey one third of my work is now done by agents." This is extremely conservative, right? Okay, so that's one third for me to dedicate on debt. Unless, and I suspect that's where companies are going, is that they maybe cannot stop themselves from constantly working exclusively on new features. "Oh, you can do much more now. Let's do more fe- more new features."
Jeff
00:21:03.465
Yeah. And they should. And they should monitor the performance of software delivery in that process to figure out if they have the right mix of debt, innovation, defects, and risk. And if they have the right mix, they're making the right decisions. Because you're right, nothing's perfect. Just watch the metrics and find the areas you gotta improve. I wanna reiterate here, I think the right people that need to understand this more now than ever before are product people
Viktor
00:21:35.494
Here's then a question because I'm going in that direction. Maybe I would put a wider net and say, product people, architects CTOs, stuff like that Are they now the right people to do the stuff? my, my question is should a traditional developer extend its role into product or should product extend its role into development?
Jeff
00:21:59.119
And should the two extend their roles into design? And should design extend their roles into product and development? And should test extend their roles into all? Yes, is the answer to all of the above. I think the team topology of today and tomorrow is one that is much smaller, less dependencies, more self-directed, more watching for outcomes, and watching the whole system. Less handoffs, more integrated work than ever before. And the reason why is because AI. AI makes any developer a part product guy and vice versa. Every product guy is now part developer. Every designer is part a- and right on through. And that's the wonderful part about it. What it means is that the handoffs that we used to have aren't gonna be sufficient. The time it takes to understand each other i- is reduced, and we all need to take into account the distribution of the kinds of work and the why between, behind what we're doing. Everyone needs to know w- a- and understand our flow performance, our flow distribution, our software delivery performance in conjunction with the kinds of features and innovation with the related value that we expect to achieve. With that in mind, developers understand the business and can be part definition of what the solution will be to that problem. Product people can be part design and part architect because they have the tools to talk to the code. And same with design. I- it's a wonderful world. Like I said, the whole world has changed and it's fantastic
Viktor
00:23:55.038
I completely agree about the won- wonderful world for myself. I'm just worried how that there is an percentage of people, whatever the percentage is, who are simply not-- don't want to be in that world. I speak all the time with developers and products and stuff like that. " " I just want to write code. That's the only thing I want to do. Tell me what to write, I'm gonna write it." Or, you know, product people, "I'm never touching the code. Never. That's not my job." Are they becoming, dinosaurs?
Jeff
00:24:31.808
Yeah. Let's talk about the world of product management for a second, just for instance, 'cause I think most people that listen to this show have, have heard a lot of development. But even on the product side, there's groups of product people whose entire jobs were to take customer feedback and synthesize it and create a backlog. There were heads of product that would create grandiose roadmaps and do that to achieve alignment working up and down the management side. There were people that listened to customer incidences and tickets and would correlate that data. There were business analysts that were part product people that really worked to understand the system and were part project managers. If you really resonate that I'm one of these roles, you should really listen because I think those roles are going to either go away or at a minimum change. The new world of product management requires that product be much more close to the customer and be closer to their correlated development partner and design. The role of product must include some key skills. You gotta know how to vibe code what you're doing. Doesn't mean writing production-ready code. It means create mock-ups that you can show to customers and get feedback, an actual signal on if what you're going to build is worthwhile. You've gotta be able to run evals on the kind of AI components that you're building so that you can decide if the quality of output's good. You have to understand how to orchestrate data and do that analysis all yourself. Meanwhile, having an umbrella hat of a product sense that you have a taste for what this product should be like. That's the new world of AI skills that AI PMs have. So now's the time to be able to retool yourself into that world or decide if you wanna go play in a different area 'cause everything else is gonna get automated
Darin
00:26:37.109
You were talking about quality of outputs. What about the quality of inputs? You talked about it earlier. It's critical now. Actually, it's always been critical. The problem has always been that we have humans interpreting what those specs and other items, those reams of paper of PRDs from Word documents that were badly formatted. Yes, those
Jeff
00:27:01.391
Those that somebody spent an inordinate amount of time creating, and then it was this great shiny document that then got put into refinement where a team broke it down into a bunch of epics and user stories, and then an architect came in and said you really sucked at that. You forgot about these 10 different things." And the number of stories and tasks tripled, and then design got involved and said "Well, this isn't the right thing." It reminds me of that picture. You remember the picture where the customer wanted a swing, and then it goes through all these different s- phases, and then at the very end it's here's what the customer actually got because it kinda all blew up. That is the problem that has to change now, and is changing. I think the world of product management of just creating a static PRD, that world is done. I am not saying that the PRD is dead. In fact, far from it. I think the requirement for product is to create a product definition that is the golden thread that lives through all these different source systems, that acts as a system that, that humans are, have an easy time inferring what would the intent was. AI doesn't know, and I- AI will happily build whatever's on paper, even if it's ambiguous, just like the example of, building a text editor. It'll just build a new one instead of being, "No, you dummy. There's source… there's platform components, there's operating system components, there, there's web component. Don't go rebuild stuff." Humans know that. AI doesn't. In fact it confidently and happily goes and builds as much code as it thinks it needs. The point is the world of product management is the driver for where this transformation has to happen, because it sits at the intersection of being able to take all these inputs, this context-rich world, correlating them together, and understanding software delivery and these different skills to build things of value, and then sit at the measurement to decide, did we build the right thing? That sounds a lot like a mini CEO. Great. That's the right thing
Viktor
00:29:05.218
there is one thing I strongly disagree with what you said, really strongly. You said humans know it. I worked with humans who build stuff that nobody wants because it was, it was in a requirement. maybe we know different humans I mean, some don't. I got the answer "But this makes no sense." "Yeah, but it's in requirements."
Jeff
00:29:26.351
You know, You're so spot on. One of the-- look I, I work for Allstacks. We're building a product targeting product managers, and its goal is to help answer exactly that question. A PRD includes the intent, not just the feature. What you're trying to do. The user stories contain the acceptance criteria, for example okay, when we're done, we want it to be able to have this set of functionality, and here's how we'll, verify or prove it. The design includes the intent of the user interaction. What happens, though, on that journey we just painted before of a PRD gets created, it's a shiny doc, it goes stale the minute it hits whatever source system it lives on, Google Drive, SharePoint, whatever, and nobody goes back to update it. Nor does anybody go back to update all the decision-making that was made since then, especially when you think of how many times were you in a cycle where you created some kind of feature, and then because of all the explosion of extra requirements and things and decisions, that the feature morphed so dramatically it doesn't even really represent what the original requirements doc was. So by the time somebody gets to it, and then a human has to ask questions like, "What, this is what we were trying to build? I'm lost." We're back to the swing problem. One of the reasons we're building the product we're building is to have an answer for exactly that question. Why, in this world of AI, do we have to be limited to just documents that are static? Why can't we ask a digital twin of the feature, maybe a digital twin of the product manager, "Why are we doing the things we do?" In fact, why can't that include evidences or receipts of the various decisions or prior work? Maybe, for example, we decided to not use a platform component and build a new one. Why would we do that? Oh, I see. Every time we tweak or have any changes to this particular platform component, it's a disaster, or it's already at scale. It can't handle anymore. Thinking of a logging component that one time we had that was a disaster. It was like, "Please, don't use it." And all those decisions could be things that people could come back to a living product definition and ask. Because we have all the ability to correlate a set of context which includes all this work and background that went into it. So I, a- anyways I'm excited about what we're building over at Allstacks for exactly that reason to have an answer to some of these things that help put together software delivery performance and a living definition of the features themselves that get put into features as they go forward, so that anyone at any point in time can say, "What did we build? Why?" Maybe after the time when we got started, we can look back and say, "Where-- Did we drift from what we originally defined? How? What did it look like? What are the upstream/downstream impacts? How could we create some kinda of knowledge base for our customers so that they know what changed, or my stakeholders in this journey? How should I evaluate success differently now from what we originally planned? How will this impact the investment?" All these things are great, and they're all part of what we're talking about
Darin
00:32:43.175
I'm going to double down on what Viktor said about the humans not being what you thought. There, there was a phrase that you'd said earlier that the developers understand the business. I can count percentage-wise on a hand of how many developers may have understood the business a little bit.
Darin
00:33:00.795
Okay but I'll give-- There are some, right? There, there are those, if you read "The Phoenix Project," there was Brent. Brent knew everything. So you're gonna have those Brents of the world. I'm gonna extend it. I have been around enough vendors that product people are coming and going almost on a yearly basis. The chances of them understanding the business is also very low. Now, in what you just sort of laid out before with the product that y'all are building at Allstacks, now we can have AI help us do that, but does it really need to help us? If we properly define how the agents should react, your digital twin, again do we need humans anymore? At some-- initially, yes. But forecast it out, not forecast it, but, looking ahead now we've trained the system in such a way that, okay we have Jeff's knowledge, we have Darin's knowledge, we have Viktor's knowledge, we have Mary's knowledge, and all of that's there. Does it matter anymore? Do, Do we need those four people or not? Or do we only need one of those four people?
Jeff
00:34:12.125
Uh, That's a great, great question, and I think there are entire companies betting on that question. Thoughtworks did a release of their new operating system for software development. There are others. Linear, for example, comes to mind in how they're approaching this same space. I'll tell you what I see currently, that when we looked at some of the token spend from some of our customers, 'cause one of the things that Allstacks has is a software engineering intelligence where think of us like an observability solution across software delivery. We create all these metrics and can show you your DORA metrics, your value stream flow metrics, and a- any other kind of slice and dice dashboard that you want. But we were looking at their token spend and saw something interesting. Initially, token spend was largely code All sorts of things that was just pent-up demand, go build this stuff. What was interesting is we started to evaluate the spend after the initial push, and after features that were backlogged, and then after let's go fix up the pipelines, and after let's bolster up some of our test harnesses and frameworks and everything else for what I would deem sort of delivery risk kinds of items for their pipelines.
The Majority Of The Spend Then Turned Into
00:35:37.675
What should we build? What's, What's it look like? How, How do we define it? What's the right thing? And starting to pull in data from the other side. Is it increasing daily active users? Are… We're starting to speak this language like uh, let's look at consumption. Did they use it? Did they click on it? Are they churning more? Are they getting value out of it? We start to have the conversation go much more cleanly to this clearly use consumption-based kind of value. Did we produce the value because people are using it? And if so, great. And what's interesting as we look back on the tokens is that's where people are starting to spend their money. Parallel to this is there's a, when we were first kicking around the idea of building Product Studio, which is another module of Allstacks, we were looking at the various competitors to Allstacks. It's okay who's targeting product people right now? And there's a bunch of tools. There's some open source ones. Even GitHub has their Spec Kit, and then somebody spun off and did OpenSpec, and a person out there did BeMan, a whole bunch of other tools. AWS has Kiro, and there's all these different tools that are out there. What's funny is the ultimate goal of these tools was to help in the process of creating agentic requirements and specs as they go forward. The token spend that they're using on generating the technical steps for the code generation is so small in comparison to all that was being spent on the requirements themselves. What do we build? Why I think that's a super interesting point when you look at the no- the token spend is because in order to answer what should we build and what's gonna get value the most, you've gotta reach a lot further upstream. What's the goal of the business? Who do we target? What's our identity? You have to have a bit of product taste. You have to understand what has worked and what hasn't as you make bets, and you need to understand where the levers are that you'll look to see if you're successful, and you're gonna make a lot of bets. So i- it's a really fun process looking at all this stuff together and seeing where this bottleneck has shifted to and where the tokens are gonna be spent, especially uh, you know, one other tangent on that, it's, I've-- Y- if you go to YouTube, you'll see no end of people building their own knowledge management system on their desktop to try to answer some of these questions, where they're pulling in such a broad set of context of stuff people pulling skills out everywhere to try to do some of these things. It's just, like I said, it, the world has changed, and it's really cool to watch it
Darin
00:38:25.085
I'm thinking about being able to take those outcomes and I'm thinking about, okay, great, let's say it's 80/20, that it's design versus implementation, We're spending 80% of our time, 80%, 80% of our token spend. These are my numbers, not yours, but let's stick with it here for… 80% of our token spend is for design and 20% is for implementation of that design How many people actually read those full designs that they're having the conversations with AI about? I'm talking reading the full thing, because if you've done a good spec, there's a near 0% chance you're going to read the whole thing
Darin
00:39:07.151
Viktor and I have talked about this because there is nothing new that anybody is creating that hasn't been created before So all of the knowledge in some way, shape, or form already exists in all of the models
Darin
00:39:23.348
It's just w- what is our… As a human, how do we have that unique selling proposition there? That was my marketing phrase for the day. I mean, That's what we've gotta try to search for if we're gonna actually make any money off of this. The on- right now, the only people I see making money are the model vendors themselves
Jeff
00:39:43.250
Yeah. Code people are people using AI to write code are making a bunch of money. There's a massive influx headed towards product people. I was just at ProductCon a few weeks back, and it was standing room only. I think people are reimagining what this organization looks like and how they'll use AI in here to make that happen, and I think the whole tooling has changed. The surface of what tools people use has changed because now I have this consistent interface that I can work with, and how do I get all the data? How do I get all the context? How do I aggregate it together and get the right stuff?
Darin
00:40:17.631
how do we get our specs ready? In theory, used to, you'd have a team of 25 people sitting around collaborating on a Word document. I didn't say Google Doc, I said a Word document uh, that they were passing around to each other on a shared drive and producing that. Now we just have a conversation with model of the day and we say yeah, that looks good. Go."
Jeff
00:40:42.148
Yeah. Isn't that funny? I actually gave a talk about this as a thought exercise of with AI, why can't we use agents to get our requirements ready? What would be the features of such an agent? What kind of capabilities does it need to pull from the requirement to know that we're doing good? And what rating scale would we use to get there? The point of it is that it's within the realm of possibilities of things that you should think about building. Build one. It's awesome. And then, of course, I said, "Oh, by the way I'm with Allstacks. We have this tool called Product Studio. We did that, and if you wanna try it out, come over here." But the thing that's interesting is why are we not thinking this way? There's nothing stopping anyone from starting to use agentic methods to evaluate requirements work, or requirements themselves, or specs or designs, or… In fact it's expected. We should. And then include humans as part of the evaluation process. describes this process of evals as having agents in the loop, humans in the loop and LLM-based analysis on evals. I think that's true for everything we do. The model's right. Gotta keep people there as a sense of reality and as a double check to make sure that the self-driving car isn't gonna park itself into something dumb that you can clearly see, but it can't
Darin
00:42:12.014
Thus the reason I will never own a Tesla. Not for other reasons. It's like that's the one reason I just like, "Nah, I don't wanna do something silly like that." You were talking about you have the platform to do all this. For people that can't buy the platform today, what are the habits they need to get into so they can get into this rhythm to where, okay, now I can have Claude helping me on this, or I'm using a mix of Claude and OpenAI to figure out, judging each other of what's going on. Is that one of the things they can do? or what do you say to some team that are saying no, we don't need any AI 'cause we got this"?
Jeff
00:42:50.764
I would say to any and every team, you will not survive without AI. it just isn't feasible. It's too expensive. You will be too slow. You're either going to have to disrupt or be disrupted, period. Second point is the cost of all stacks is such that you just give it a try. It's gonna make you faster, better, stronger, all the $6 million man things, if you remember that reference from way back in the day. there's no reason to not. And if you want to try it out yourself, just go to your own Claude Code instance and tell it to write you up a little agent to validate your specs, validate your requirements, see how well it does, and then you'll create an appetite where all of a sudden you're gonna want some things. If you haven't read Andrew Karpathy's little blog about all the different things that he's doing, especially the create your own knowledge management system and put Claude on top of it to evaluate your conversations and to track the things you researched. It's a great exercise that will change the way you work. I started on that path, I don't know, some time ago. I take notes completely differently now. I operate differently. My personal to-do management is completely changed, all because AI. Side note on that, which is funny. I'm a yes guy. If I'm in a conversation, as you can tell I, I like to talk, and I say, "Yes, I'll do that," to a lot of things. And then I would promptly forget about a third of them and be like, "Oh, I need to remember to do that." Part of my system of AI changed that. So a lot of the meetings I'm in are recorded, and so I track those. I pull them down, and I have AI every day pull out, "Here's the to-dos I'm supposed to do," because this is what I committed to. Like, "Oh, yeah," and I go get it done. I've actually taken that the next step of where it's helping me pre-create a lot of the tasks. If it's an email, it'll create the draft and so forth. So this is our new work style
Darin
00:44:57.208
I'm thinking back to Allstacks, maybe not, because I haven't looked at the products, But initially you're talking about, hey, we've got all the DORA things, we've got all the engineering metrics, blah, blah, blah, blah, blah, all the things. Do those even matter anymore other than token spend? Seriously,
Jeff
00:45:14.145
No, I love your question. Gartner seems to think so because they just did their Developer Productivity Insights Platform Magic Quadrant, of which uh, Allstacks was in and was a visionary. Uh, v- yeah, I know. Yay.
Jeff
00:45:30.365
Bless their hearts, exactly. But I've been saying that for a long time. Does developer productivity really matter anymore? I mean, uh, We wanna… How do we measure productivity? Do we…
Jeff
00:45:42.845
Y- right. to that point uh, we take the really the shortest period of time in the overall cycle, and we're gonna measure them more than any other organization has ever been done, more telemetry data on engineering than any other org. And yet it never really was the biggest problem. It was a problem, but not the biggest problem
Jeff
00:46:06.580
I still back believe getting people aligned and the definition side and breaking this down so that teams operate more autonomously versus trying to achieve some big standard where everything operates at a fixed rate across a fixed kind of boundary of time, that we must have some kind of big room planning and manage your dependencies till the cows come home. I, I so hate all of that. You lose the plot. Unleash people unleash people. Create mini product team. I contend that the strongest transformation in any team isn't necessarily AI itself. It's creating a completely autonomous, self-directed mini CEO over a feature who's gonna whip the results that he was looking for through the team itself and making sure that the team is aligned in what they're building. They will have to use AI to get there, and he will prove the results. The minute that a company turns into being more of a, like a venture capitalist and operates in that model where money gets doled out based upon business performance of each individual feature and team, great. Side tangent. I still remember, like I used to work on the office team. We built a little product. We had to go to quarterly reviews with they were our Bill G reviews, and either you were on track to be $100 million business or, find a different group to work in or different company. Go make it happen. I love that model. Internal teams would compete, build different things. Okay. See who's faster, see who's realizing value, see who's gelled. But put it on product to go build that out. Put it on product to go be the head of that feature and own their destiny. 'Cause then they're impassioned, it's their baby
Darin
00:48:00.837
Values, outcomes, all the things that we should have been measuring all along, but instead we got busy measuring the things that we thought we could measure to begin with, like lines of code. Give me a break. if people are listening today, what's one thing you'd want them to remember from today if they don't remember anything else?
Jeff
00:48:18.918
The one thing to take away from this is there's not some big transition that you need to go to school for, spend three years, and you're going to come away being all new. This is a different style. Start taking the tools today. Start working today. Do one little change every single day, sort of Atomic Habits-esque. Be 1% better tomorrow, 1% closer to being self-driven, understanding the goal, breaking dependencies, using tooling, keeping up with what's changing, and looking ultimately at the value that's being delivered. in the process, just a little bit of improvement every cycle, every week, something a little bit better. There's no reason to not, even for personal use. Experiment, play, tinker
Darin
00:49:12.365
20 bucks a month. W- and then once you get started on 20, you'll head towards 100. Ask me how I know. I haven't hit the 200 yet, but I could. Tell us more about Allstacks. What is… You know, you've talked about different features. What is Allstacks as it stands today in September of 2026?
Jeff
00:49:32.093
Allstacks has three core modules. We got our start in the engineering intelligence developer productivity space. So if you want a bunch of dashboards and a bunch of metrics about anything and everything, great, we got that. There's also an MCP server so you can pull those metrics into your own whatever you want to visualize kind of engine. That's interesting and required as a context graph as a set of data so that you can do other stuff. What's that other stuff? We enrich that context graph with capitalization and investment hours, meaning we give timesheet-level fidelity data to every person that's working on the team. We know what they're doing. We know their activities. We see where they're writing code, writing docs, writing designs, and can track their usage by correlating source systems to the things that they check in and understand how much time is spent. The big thing that Allstacks is doing is we, in, late May, released a product called Product Studio that's targeting product managers. And Product Studio's goal is to help create that living definition of a set of requirements and the charted out work tree and integrate with the prototyping tools and create a set of documents that are living with a chat interface so you can ask w- why a feature is built the way it was built, why it has a particular design or used whatever components it did, or what customers were looking for things, and have that remain and accelerate so that it produces human and agentic-ready code. So it's what we like to call build-ready or we're shovel-ready on once you get done with things. And we have a bunch of other stuff that we do in the back end from uh, agentic risk analysis, agentic requirements readiness workflow analysis, looking at those balance patterns and so forth, and a lot of other things we put in. The speccing process includes things like adversarial reviewers and things that include various formats of the documents that get put out and different systems that we integrate to. But ultimately, the whole goal is to help accelerate the process and method of building the requirements by having an expansive context that's managed by one system versus individuals trying to build this themselves. Find that product leaders are just, have so much on their plate now. If developers produce that much more, their pressures are just massive to hold all this context in their head and produce more and manage the results. It's a big job, so we're helping with that.
Darin
00:52:13.929
So you can find Allstacks at allstacks.com. That's A-L-L-S-T-A-C-K-S.com. All of Jeff's contact information will be down in the episode description. One question before I let you go. You said you worked on Microsoft Office. Please tell me you worked on Clippy
Jeff
00:52:30.852
No, I uh, sadly I did not. It was a really fun time. We shipped a product in the Microsoft Outlook box so, uh, such a fun time
Jeff
00:52:44.600
Thank you much We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox.com/contact. If you subscribe through Apple Podcast, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox