Alex
00:00:00.000
So it's more of a configuration management problem in that context, and that doesn't really give you what I was referring to before as far as runtime control and being able to, flip a flag in production in seconds. If flipping a flag still requires a full CI/CD pipeline and a full redeployment, is it really a feature flag? In our opinionated view as Unleash, we don't think that's the way to use feature flags because you can… That's more like an environment variable, right? That you can configure, the application reads it at startup time, and that's it.
Darin
00:01:36.949
There's a new word floating around. We've got new words floating around all the time. This one is FeatureOps. At least they didn't try to make it DevSecFin FeatureOps, because that would've just been more silly. Is FeatureOps really a real discipline or not? Or is it just a marketing wrapper around yet another thing? Today we're gonna be talking with Alex Casalboni from Unleash. Alex, how you doing?
Alex
00:02:02.588
Hey, guys. Glad to be here and thanks for inviting. And yeah, let's talk about FutureOps today
Darin
00:02:06.995
Let's talk about FeatureOps because there is a site for FeatureOps and just we'll go ahead and tell you up front because it's featureops.io. Feels very agile manifesto to me. Is that sort of what you were going for? Is like, " here are the 95 theses of how features should be used"?
Alex
00:02:24.720
Pretty much. That's, based on what we see with many of our customers, but also many of the experience of, the founders of Unleash and many other people that work at Unleash today. And the main point is to be, super vendor neutral, super, referenceable because it's a problem that basically every company has, whether you are, in a financial industry or a big retailer or, whatever you're doing. Managing, feature management and release management is a big problem, and handling that when you have 1,000 engineers across, 25 teams at scale, and especially when, if you're down for an hour, it's millions of dollars and reputational damage and all those issues. You know, How do you handle this process at scale? That, that's what the manifesto is for really.
Viktor
00:03:14.478
I feel that we need to backtrack for a moment for this feature ops, ' cause I was-- before you started speaking I was imagining my PRDs that I passed to my agent, this is the feature I want. I'm assuming that's not where this is going
Alex
00:03:29.790
Yeah let's do a step back. Let's take a step back. I think the main… Fir- first of all, why ops? Because there are some operations involved, and most of the other discipline, like DevOps and all the cousins pretty much help you until the deployment. You can do all the things in the local environment. You can do all your CI/CD. You can do all the static check and all the GitHub's magic to make sure the environment is aligned to your configuration, blah, blah, blah. But basically, a lot of the, you know, 90% of the approach is you ship to production, and then if something happens in the production environment, you need to go through another cycle of that. You need to go through a hotfix. You need to go through another CI/CD pipeline, running for 20, 30, 60 minutes. It depends. And so what the FeatureOps methodology is trying to address is can you apply the same principles but after the deployment is done, when the application is running in production? And the better term to understand what I mean is runtime control. Can you change application behavior at runtime in production to reduce the blast radius of an incident? Things like a kill switch, things like something that you now, that you can enable or disable or change the behavior in seconds instead of waiting for a patch from the development team and then a full CI/CD pipeline running for half an hour, and then you get the fix in production. that's like, the main reasoning behind core
Darin
00:05:09.593
So what I really lacked when I was setting this up is we're talking about feature flags, but from a little bit different angle What we're used to with feature flags, and we've talked about feature flags on this show many times, is, okay, back in the old days it was kill switch, right? Or turn something on, turn something off. But then it became more important. We also have to do maintenance of those flags 'cause flags eventually go stale, become useless. And that's really, to me, sort of the part of it is the, governance and compliance part of this is, how do we deal with turning those flags on? It's like y- it's easy to turn a flag on, You click a box, you run a CLI, you do something, flag's on. But how do you make sure that you have the 75 different approvals to turn the flag on?
Alex
00:06:02.178
it is hard. It's hard. I think we- if you look at this problem space, you can look at it from three different angles. One is the technical angle, like yeah, how do you do the thing? How do you integrate the tooling into the product? How do you make sure it's up and running and you have great availability and all those technical things? The second angle is the people. How do you coordinate people? How do you handle, a role-based permissions? How do you g- give access to the right people, to the right, environments and applications and projects and whatever you have? And the third angle is compliance and regulations, and how do you make sure that this all lands well in your, you know, risk management processes and all that fun stuff that enterprises need. So we can look at it from these three angles and talk about what's actually hard in these three spaces. I think what you were referring to mainly is, the second, like the, how do you coordinate, how do you have processes around cleaning up after, a release is actually completed, and how do you coordinate all of these effort in a, maybe in a centralized place in like, uh, maybe you're a platform team and you can enable all the development teams to use this thing as a centralized service instead of everybody reinventing the wheel. Like, There are many conversations across these three angles and, w- which one you think is, you know, more relevant or more challenging?
Darin
00:07:35.390
They're all challenging, and we need to solve all three with- within the next 45 minutes. So how do we solve that?
Alex
00:07:42.016
first of all, there are different ways to do feature flagging. I've talked to many people that only think of flags as something that lives in their configuration, which means it's maybe read statically when the application starts up, and then it doesn't really change ever. So it's more of a configuration management problem in that context, and that doesn't really give you what I was referring to before as far as runtime control and being able to, flip a flag in production in seconds. If flipping a flag still requires a full CI/CD pipeline and a full redeployment, is it really a feature flag? In our opinionated view as Unleash, we don't think that's the way to use feature flags because you can… That's more like an environment variable, right? That you can configure, the application reads it at startup time, and that's it. it But it doesn't give you the runtime control that a proper, feature management solution should give you. And then some people actually do it at build time. You know, I was reading some articles from, the companies of the decade, like Anthropic and some of the big ones that are now, pushing the boundaries. And for example, they do it at build time. They read a flag. If the feature is not enabled, they don't even include the code in the CLI bundle or stuff like that. So that approach to feature flagging is, build time or startup time. It's not about runtime control. And there are reasons for that, right? You want to reduce the bundle size. You want to not release unfinished code into a CLI that people might reverse engineer and blah, blah, blah. That makes a lot of sense. But what we see when we work with companies every day is that, enterprise companies especially every day, is that they need a mechanism to toggle stuff in production in seconds because you… It's not probably going to surprise you, but many companies have like, a 12 to 24-hour round trip between when they discover a problem and when the fix can go to production, doing the full release cycle even for a hot, for a hotfix. So it's a big problem.
Viktor
00:09:47.290
If you're talking about seconds, to feature something, we are really assuming that you're talking about total duration, not seconds after I click a button. we're talking
Alex
00:09:58.991
and between when you click the button and when the behavior in production changes. That's what I mean
Viktor
00:10:16.009
Okay, it's been a day since we deployed this. A day later we discovered it doesn't work. We click a button and two seconds later it's fixed. That's one day and three seconds duration.
Alex
00:10:30.399
But usually you measure between when you find the problem and when you can release a solution to the problem, right? And so if you find the problem 24 hours later, some companies will have a fix in production for the problem another 24 hours later, so it's two days. So I think, yeah. Does it make sense?
Alex
00:10:54.687
For some companies, it's six hours. For some other companies, it's one hour because you're smaller and leaner and you're more like a small, medium business or a startup, and you can push a hotfix in minutes or like less than an hour between when you find and identify the problem and when you can push the fix to production
Viktor
00:11:12.836
Where I'm going with this question is basically if you measure the time for real, not from the point that is convenient after pre- clicking a button, then the real bottleneck is how do we detect an issue and fix that issue right away? I'm talking metrics, I'm talking traces, I'm talking logs, and so on and so forth. So I'm assuming that feature toggles of some form or another once we've solved the button problem, they must be solving that problem in a similar way that canary deployments are solving it, but through in case of canary deployments, that would be rolling back the whole binary, right? And in case of feature flags, you're rolling back a feature. And then you can actually talk about seconds and days plus seconds.
Alex
00:12:04.122
I think most companies measure mean time to remediation or mean time to whatever is, again, the mean time from when you find the problem, not from when you deploy the problem the last time. But that's not a problem. How do you find the problems as early as possible? That's not a problem that feature flags could or could not help you. But to your point, yes, there is an advanced, let's call it more advanced way to use feature flags in the sense that they can act more as a automated mechanism, similar to an automated kill switch or a circuit breaker, where you're not just waiting for a human to toggle the flag, but the feature flag system itself knows what's going on. We call this impact metrics. Basically, what you can do with the Unleash SDK is that not only you use it to check whether a feature is on and off, but you also use the SDK to send back metrics to the system so that if error rate crosses a threshold or some other CPU, whatever, crosses a defined threshold, you can automatically switch off that feature without a human intervention. Because you're right, if you look at this from the second angle we were talking about earlier, usually the bottleneck is not the clicking a button. The bottleneck is, okay, I figured out there is a problem, but what can I actually do to remediate it as quickly as possible? If it takes a full cycle, it might take hours. But still, if it takes 25 million approval checks and a lot of internal bureaucracy and all those things, there is still a process or bureaucracy or human bottleneck between when you know what to do and when you can actually do it. That's exactly where we are investing a lot on automating this concept too and integrating the business metrics or the technical metrics into the system to do the work for you in seconds again.
Viktor
00:14:05.510
But if the problem is not technical, then the solution is not technical, right? You said that we have a bunch of approvals and what's not right. Those are not technical problems.
Alex
00:14:16.552
It's uh, often a compliance problem 'cause a lot of large companies need to track everything, 'cause flipping a flag in production is a change in production, and a lot of regulations require you to have audit logs and to know who decided what and when, and what exactly changed at what time. So what we see with our customers that maybe they have like, an in-house feature flagging system that doesn't really provide that auditability, 'cause everybody can go to the database and run a query and flip a flag in production, and that's a big compliance black hole because, how do you prove what happened, who did the query at what time, what exactly changed and when? And so th- that's one of the enterprise features that we definitely provide at Unleash where you have an immutable, event log that serves you as a auditing mechanism which includes if things happen automatically, or if you have a sophisticated change request process where you need a four eyes principle and you need at least one or two more people to approve the change in specific environments like production. So absolutely, yes. It's not only a technical problem. That's what we were talking about at the beginning, where technically it's, everything is possible and, the SDKs and some of the features help you there. It's also a people pro- problem 'cause, sometimes it's not an engineer that wants to flip a flag. Sometime it's a product manager. Sometimes it's, a release manager in the bigger companies. And it's also a compliance problem because, you, you have your ticketing system, you're using ServiceNow or whatnot, and you need to make sure that everything makes it there as well, and, you know, there are ways to automate that part as well.
Viktor
00:15:58.385
it's a bit confusing because Do companies assume that all toggles that they have are safe to toggle, or they make assumption that actually it's not safe to toggle some toggles? Because I can only assume it's latter, right? Because if they would say it's those toggles are safe to change, they wouldn't be asking for approvals. So basically you're saying that they treat every toggle as essentially a new release. Not from the duration perspective
Alex
00:16:34.925
and that's exactly the point of FeatureOps. The main advantage you get from this approach is that you can finally decouple the act of deploying the application from the act of releasing a feature to your users. You know, You can deploy your code anytime, multiple times a day, automated, with your continuous delivery, continuous deployment. But that doesn't mean you are releasing the features when that happens. that's a business deci- decision that should happen when the business and the customers are ready to actually start using the feature. To your point, yes that's the major difference, and it's not only when, it's also who decides. As an engineer, you have written the code, you have written the functionality, you have written the whatever, but it's often not you as the engineer who decides, "Okay, we are launching this next Tuesday at 5:00 in the a- morning or 6:00 in the afternoon."
Viktor
00:17:27.403
And in that case, you don't need approvals except the person who is a single person, right? Because it's not anymore… We have approvals because we don't know whether it's safe, but now we're saying this is safe by definition,
Alex
00:17:42.300
Le- let's talk about what a feature flag can do because there are many ways to use it, right? If it's like a release flag where like you are deploying a new feature and you have to decide when to enable that new feature, that's something. But it could be a kill switch that disables an API call or disables a third-party integration, or it could be a permission toggle that decides who can access what because of their subscription tier or something like that. it could be an A/B testing experiment where the feature value is not just true or false, but the feature value is, test A, test B, or test C, and there are different things you're showing or different things you're doing, like you're A/B testing an AI model or you are A/B testing a visual functionality or the copy on your homepage. So it really depends. Is it safe? What is the toggling act gonna do in production? And that's part of your feature ops discipline, like who is doing what, who is responsible for what, and also what's the outcome of toggling that flag. That, That's super important, and you can have internal documentation. The feature flagging platform can somehow help you with that because, for example, Unleash has different feature flag types that you can keep track of, and you have descriptions, and you have all the things that help you understand what happens when you toggle that flag. But to your point, a lot of organizations just need approval 100% of the time or some kind of change request mechanism because that's a change in production, and any change in production needs to be tracked and need to be auditable and needs to have the four eyes principle applied so no single human can say, "This is changing in production now because I decide it." Somebody needs to review it and approve it. Doesn't mean that every flag is safe or unsafe to toggle. That really depends
Darin
00:19:33.280
Let's stick with technical for just a minute. You said something a few moments ago, immutable event log. I have to call BS on that. There is no such thing as immutable data.
Alex
00:19:43.100
Fair. I'm not going into the blockchain world 'cause I'm not an expert there, but that's not what we're using, so I'm with you there. The idea is data that nobody has explicit access to, like in a easy, direct way, whether it's like a lockdown table somewhere or some object storage where the application and the keys you are delivering only have read… Sorry, only had write access to in append only mode. Whatever implementation, that's a technical detail, but or whether it's a ticketing system somewhere, that's what most compliance frameworks require, that there is an auditable ticket or Slack conversation or some-something in ServiceNow that shows who requested what, who was the approver, and when the change actually happened. Can it be changed or tampered after? Yes, and that's why there are a lot of products that try to help you with, you know, avoiding that. But yeah, that's something that we provide at Unleash. I'm lacking the internals details about how that is implemented, but I'm happy to share some details later
Darin
00:20:47.444
No, that's fine. It's because I hear immutable just spouted by people and it's like, now hang on a second. To an extent, blockchain could also be become mutable as well, but it-- that would take a lot harder work versus what you were just saying, "Oh, something goes into a bucket." Well, I can change the keys and get access to the bucket, so that's not immutable. We were talking about lots of other acronyms too today. In the US we've got SOX and FedRAMP, but in Europe, y'all have something called DORA. No, not that Dora, but the other DORA, the Digital Operations Resilience Act. W- is that causing more problems than help, as most regulations do?
Alex
00:21:25.358
I'm not an advisor here. I know that DORA mostly applies to financial institutions if I recall correctly. I remember it does apply to, resiliency and making sure, your application, your infrastructure, your dependencies are resilient, blah, blah, blah. I don't have enough knowledge of the details here. I don't know if it's causing more trouble than actually, helping companies. But it's not the only regulation and it's all, there is GDPR, there, there are many more. There's NIST. There, there is more. And I think the important thing to remember, if you are a technical guy like me or like us trying to navigate this space, is that what's on you is usually the technical implementation and the tracking and making sure, that the thing is resilient. I'll give you an example. A lot of feature flagging, whether systems, whether they are in-house built, a lot of companies just build in-house 'cause sometimes makes sense or like an external service, they rely on external API calls for the actual evaluation of the feature flag. Now that has two main issues. One is obvious, one is performance, because if your server-side application or your front end has a dependency on an external API call, and maybe you need to make three, four, five, or 10 of them to render a page or to serve an API request, that means additional latency, and that's a problem 'cause, a round trip back and forth to any API will almost never take less than 10 or 20 milliseconds. And if you do it 10 times, that's a lot of additional latency for just checking a toggle. But the other problem that maybe some engineers wouldn't think about immediately is the data privacy problem. Because most feature flags, yes, they give you a true/false or some kind of variation if it's an A/B testing as an output, but we almost never talk about what's the input of the evaluation. The input is not just the application name. Usually, the input is the user context, because a flag can be true for a user in France and off for a user in the United States, or a flag can be on for a user that has a premium subscription the targeting rules of your feature flagging system will usually have some targeting logic that depends on the user attributes or some kind of context. That's the keyword here. And so what's included in that context and where that context needs to be in order for the feature to be evaluated that affects your privacy and your compliance, right? If you need to send all the user context, including possibly, PII and personal data and all those sort of, you know, sensitive information in general, that means, you are sending the data out of your perimeter, out of your infrastructure, and that's a big problem. So that's why most mature, robust, resilient feature flagging solutions have some sort of local evaluation mechanism where, first of all, you don't need to do an API call every time you check a flag. You get some configuration at startup time, and then you keep it fresh every few seconds or minutes, and the evaluation happens with some local process, local logic. And for the same, with the same mechanism, the data stays where the application is running. You don't need to send the context out. So that's why one of the examples for, if you're building it in-house, for example, that can get really tricky because, you know, maybe you start simple. You don't want to add a lot of complexity to your SDK or to your local evaluation logic, but that means you're adding latency and you're sending your data out somewhere. Maybe it's not a problem at the beginning 'cause everything's running in the US in your data center or on AWS or somewhere, but then you scale out and you add additional regions and maybe you don't also scale out the service, so the data needs to go back to the US if the app is running in Europe, and then that creates problems, right? So yeah, that is one of the issues we help our customer with from the technical and compliance angle.
Darin
00:25:38.799
I'm thinking about one of the things you said there. You brought up config, and I'm gonna sort of twist it a little bit. To me, it becomes a smell when a feature flag basically becomes a permanent configuration switch. pricing tiers, customer entitlements, things that were potentially A, B or A, B, C, D, E, F things and you've picked one. Is that really a problem? Am I just overthinking it? Or is that just the next evolution of what flags were meant to be?
Alex
00:26:06.810
I think it's real and there is a lot of evidence around this and also previous incidents that prove that if you have too many stale flags over time, the technical debt really adds up and compounds because it adds more overhead to every, code review. It adds more complexity to adding new features if, there are too many if uh, switch or different brands to even think about in your brain. that's something that, in my opinion, every feature flagging solution needs to help you with which means for… I'll give you an example. When you are on Unleash and you go through the, you know, setting up the flag, integrating it with your, the code and then switching on in production, at some point there is a time in the future where you can call this release or this experiment concluded and Unleash gives you a wonderful dashboard that shows you like your instance technical debt. And so it helps you, okay, we have seen this feature flag in production for a few weeks and then it stopped being evaluated or it's always evaluated as, true 100% of the time so it's probably ready to be cleaned up. And we have also run a few POCs with some customers where, you know, when you click mark as completed on Unleash, you spin up an automated process that will go and clean up your code automatically because you have the feature flag name, you know what's the outcome of that experiment or that release. And for example, we have an MCP server that will help you automate the cleanup process based on the specific language, the specific framework. And so what we set up with this customer is basically a, GitHub action kind of workflow that will automatically create a PR for you using the Unleash MCP server to clean up your code of a released feature flag. I think that's one of the ways you can tackle the technical debt problem where, you don't have to wait 10 years. I was talking to a customer a couple months ago and they told me, "Yeah, we want to migrate from our in-house feature flagging solution to something and the oldest feature flag we have is from like 2012." Yeah, that, that shouldn't happen and we want to help you with that.
Viktor
00:28:22.547
Do we have this… You just mentioned MCP, right? Agents, what so not. Do we have the same problem now with feature flags cleanups as we have with reviews that, okay, now whatever you had before, now it's multiplied by infinity, right? Now i- for every pull request you got, you had before, now you have 50. Is it the same thing with feature flags, multiplied by 50?
Alex
00:28:46.445
It depends. how you think about it and how quick you are with cleaning up. What we suggest is to have a very intentional internal guidelines about what requires a feature flag, what domains, what subfolders in your app, to put it simply, require a feature flag, whatever is changing in that folder, for example. But not everything requires a feature flag. Like, If your default is every change requires a feature flag, it definitely becomes a problem. For example, what our NCP server provides is a bunch of tools that are not just API calls. It's not just an API wrapper. It's domain-specific guidance that helps the agent do something that it wasn't trained to do. Like most LLMs were not trained to do feature flag getting or some of this stuff. So what the NCP gives the agent is opinionated guidance about what to do, and you can, of course, customize the logic. there is, for example, a evaluate risk tool. It doesn't do any API call, but it does take a look at your code base. It does take a look at the change that's current in your Git history and, whatever you're doing in the code. And it can give you an evaluation of a risk level of the change that's, currently being developed. And that's one of the things that you need as a developer to understand, you know, to speed up and to even automate the process. That's something we see… Because to your point, a lot of people now are using LLMs to write their code. And so unless you integrate all of these feature ops methodology that we're talking about into their, Claude or whatever best practice that they're using it just becomes an afterthought, and that, that's also a problem. But yeah, review fatigue is real. I don't want to deny it. I don't think feature flags are, specifically solving that problem. But if you can automate the building the feature flagged feature automatically, and if you can automate the cleaning up afterwards, you are already halfway there, and you don't end up with 10 years of technical debt to clean up
Viktor
00:30:50.684
Does any of the users you work with do automated cleanup kind of like I, I trust it? if it's true for a year, just don't ask me questions
Alex
00:31:02.784
short answer is yes. Longer answer is, if you asked me two, three years ago, people had static analysis tools. People would do r- you know, regex matching, trying to find the things to clean up in the code and apply static rules. Now i- I think, in my opinion it has become easier 'cause, an AI coding assistant already understands your code base enough to be able to understand what to remove, how to clean up, how to clean up the imports, and how to clean up some tests that are not needed anymore. You know, Not just the if logic. So based on my personal experiments and the POCs and the things we are running with our customers that is now a lot easier. And so you don't really have an excuse anymore for having 10-year-old or 5-year-old or even 10-months-old flag in your code base because you're just making the life of your developers harder without no real reason
Darin
00:31:57.679
If you had to guess a number, what do you-- percentage do you think in a customer's system, pick an app, pick a whole system, of the flags that are there are actually stale? 50, 60, 70%? Is that higher than we want to admit?
Alex
00:32:11.872
That's a great question. Thank you. And I think I did see a statistic recently from a research paper. If I remember the statistic correctly I'll correct myself in maybe in the notes somewhere if it's wrong. But the statistic I remember is that there is an order of magnitude difference between how many flags a company creates every year and how many flag they clean up every year. So I don't know the percentage, but if you create 10 flag and only clean up one, over time, that, yeah, that creates a lot of troubles. And that, yeah, if I had to be honest, that's what like, the main pushback that people have about feature flags. "Oh, it complicates the code." Everything complicates your code. It-- The best way to not complicate your code is to not code But uh, if you need it, we want to help you make sure it doesn't complicate your code in the long term. So let's collaborate there
Darin
00:33:04.962
I'm going to go back to using AI and the MCP server and stuff to rip things out safely. again, I don't trust it. excuse me. I use AI on a daily basis. If I could have a hourly basis, almost a minutely basis. Is that a word? I trust the AI. I don't trust, especially knowing flags and having worked with flags since the early days, I don't trust it because I know the Cartesian of flags. It's like, okay, we're gonna take this one flag out. Okay, that one flag down a happy path, okay, that will be easy. It was a simple kill switch. Get rid of it, because that kill switch was for a system that we no longer integrate with. Great. That one's easy. The problem is when we get back to the, okay, this is one flag, going back to your user context not AI context, but actually, okay, what amount of data do we need in order to decide if this flag needs to be on or off? I don't know how we deal with that behavioral side of things. It just doesn't seem safe to me
Alex
00:34:03.072
To me, I would describe that as discipline. If it's about tribal knowledge, like yeah, that team, that engineer knows, only that team and that engineer knows what this feature is doing in production. You are counting on avoiding human error. While if you try to encode those rule and that knowledge into playbooks and now, AI agent documentation and, and prompts and things that can help automate some of this stuff as well, I think that becomes a bit easier 'cause you're not counting on, the 12-year-old flag that's still there. It's probably 'cause somebody left and nobody wants to touch it, right? That is a typical problem of companies that have a lot of legacy stuff around. And so how do you solve it, is usually a knowledge-sharing issue or a, documenting things issue. I don't think AI is less trustable in that sense. I think the potential of solving the problem is more is higher than the risk of doing it with AI instead of doing it manually. But, this gets religious really quickly and political really quickly so, I completely understand if you don't trust, doing some of this stuff with AI. And you shouldn't trust, you should verify anyways. That's another argument, right? Don't just trust it
Viktor
00:35:20.926
is there a collision between feature flags and canary deployments in terms that do we need both?
Alex
00:35:28.275
That's a good question that I talk about this with m- some of my colleagues almost every day, so thank you for asking. So first of all, there are situations where you need both. But if you use canary deployment for enabling or disabling things, like similar logic to a feature flag, I think you can avoid it, especially if, you know, doing a canary often means or like a canary or blue-green or other techniques that are similar, it often requires that you have, different clusters or different environments running, and you can switch them easily at DNS level or some o- some other technique. And so if you can do that, canary or blue-green kind of logic directly in the application, it means you can do everything in one environment directly in production. You need to do a lot of switching and… But also, it's about the complexity of what you can do. If you do a blue-green or a canary, usually it's about one thing, and then when you do the switch, you're switching one thing. If you want to have three A/B tests running and 10 different flags or, behaviors that you want to change, that's really become too complicated to do with a canary or with a blue-green deployment. So my sh- shorter answer, sorry for digressing, but my shorter answer is, if it's about infra stuff that doesn't require using user context, that doesn't require, you know, application-level knowledge, but it's just something that you're switching, like you're changing database, you are switching to another API vendor, you're switching to another host name or something, that still makes a lot of sense to do things at the infrastructure level, you know, with your canaries and your blue-green deployments. But if you are using those techniques for application-level, flagging or behavioral changes or feature flagging, you know, I think feature flags are a much better tool. So you can use them at the same time. The reality is that different teams at different layers of the stack will probably use different techniques
Viktor
00:37:23.667
does that mean that everything I'm coding needs to take into the account all the permutations of the current code? So let's say that I'm building a new feature and there are already 10 features that are under feature flags that's I don't know, 10 to uh, 100 permutations already, That my new code, new… My new code essentially needs to have 100 feature flags in itself
Alex
00:37:54.065
That's the theory. Like combinatorially, it gets really complicated really fast. The reality is that most teams try to test… First of all, if we assume, and that's a good reasonable assumption in my opinion, that a feature flag shouldn't stay there for 12 years, and that if you're doing a release, that will take a few weeks and not a few months or a few years. Which means once something is released and you decide that the feature flag is on for everybody, after three, four, six, 12 months, you can clean that up, and you really have no good reason for switching that off after the new functionality is available to everybody and not just a sub-segment. So I wanted to start there because, if you assume everything can be toggled on or off at all times, yeah, that's a problematic assumption. I think a more reasonable assumption is some feature flags only go from off to on and then stay on forever or until you clean them up. That's like the typical workflow of a feature flag. Some others, like a kill switch, they will stay off most of their life, and you only switch them on for an incident or for a specific situation where you need to kill that integration, you need to kill that code execution path. So tho- those are the two extremes, and then there are some shades in the, in between. But so the reality, long story short, is no, you don't need to che- to check all the possible combinations of all the feature flags. But there are some that are more likely to happen. So what you want to start from is what's the current state in production? So maybe you have 10 flags uh, seven have been on for the last three months. One is a kill switch that's always been off, never been turned on unless you actually need it. And a couple of those are ongoing A/B testing experiments that are in like 50% rollout or whatever. And so yeah, you can test, the major from the production state, what happens if you toggle on and off any of those, and then try to test, the major branches, not all the branches combinatorially. So I don't give you-- I wanna give you a number, but if you have 10 flags, you can probably test five or six cases, not 100 reasonably. But yeah, there are also ways to be a bit more strict with this and with the numbers and with the math.
Alex
00:40:20.641
I just made an example. Because the practice is you don't have 10 flags. The practice is customers usually have hundreds, and some of those are, you know, land in different categories and have different history. it's also true that most flags explicitly never interact with each other, meaning they do not cover the same code path. And so if there is a situation where a flag depends on another flag, there are also ways to, for example, do uh, hierarchies or, you know, parent-child relationships. That's something you can do in Unleash. So for example, you wanna make sure that a flag can only be turned on and off if another flag is on that's like his parent. So you can handle those dependency explicitly in the tool that you're using without counting on, again, documentation or tribal knowledge or humans remembering what are the dependencies. In the general case, though, I think it's fair to say that it's a best practice to avoid feature flagging interacting with each other. I do remember some white papers saying that even if you do it, there are still chances that some feature flags will interact with each other, and so you need to remember it, you need to document it, or you need feature flagging platforms that allow you to track and link and create your relationships with- between flags, so the platform is aware of it and warns you if you try to do something wrong
Darin
00:41:43.346
We've had flags and experiments for-- We had flags longer than experiments, but the whole feature flag as a marketing thing 15, 20 years. It's now 2026. What's new in feature flags in general? Just, it still feels like it's the same thing over and over again.
Alex
00:42:02.022
What's new in Feature Flag? That's, That's an interesting question. I think one thing that, the thing that we talked about at the beginning where that's what we call autonomous feature management, like the idea that you can feed actual production data into the system so that the system knows if something's wrong and can do the work for you of enabling a kill switch and doing automatic circuit breakers or automatic pausing a rollout if the current cohort is experiencing high error rates and things like that. Because yes, to, Viktor's point half an hour ago, if the bottleneck are the humans realizing there is a problem and figuring out which flag needs to be toggled off, and if that takes hours, you have still hours of issues in production. So can the feature flagging system help you automate that? So you can call it automated or autonomous. We like autonomous 'cause it's a bit more AI adjacent, as you can imagine. But the idea is can you build a smarter system that helps you remove, the human or the bureaucracy or the procedures bottleneck from, how long it takes to remediate an incident in production?
Darin
00:43:15.397
What's one thing or a couple things that you wish people would stop doing with feature flags?
Alex
00:43:21.547
Ah, that's a good question. It might be a repetition because we talked about this a lot in the last hour or so, but treating feature flags as something that you can forget after you ship the code. that's a main challenge that our largest customers face, where you have 1,000 engineers and they all now push tens of PRs per day as you do. And then they forget that there are now a billion feature flags in the system that needs to be cleaned up. So what should they stop doing or what should they start doing is stop forgetting about it, stop treating it as somebody else's problem in the future, and start including the cleanup process into your definition of done so that, in a couple of weeks, either you automate it or you remember because there is a ticket about it, and you go and clean up whatever needs to be cleaned up. And we can help you with that, with, reminders and dashboards and automated workflows that start when you mark something as completed. But uh, it's still part of your internal culture and processes to just not forget about it and not think that somebody in the future will take care of it. It's still your responsibility as a developer or as a product manager or as a release manager to take care of the cleanup. So plea-please do, and please let us help you tackle that
Darin
00:44:44.412
I think the next feature that you do need to build within your product is a self-destructing feature flag. Sort of Like "Mission Impossible," you know? you listen to the message and its message is over and now it's gone.
Darin
00:44:59.692
yeah, you're welcome. Yeah, you… That, that'd be a $5,000 recurring fee every time you use it. So we don't want people to forget that they're there. That's the problem. Couldn't we argue that if there's such a pain and now we have billions, your word a moment ago, of flags in the system, why do we even bother doing flags at all?
Alex
00:45:18.336
That's a good pushback. I think I, I have this pushback many times in the last three months, and usually it's from someone that is telling me, "Okay, but I've been trying to shift left everything in the last 10, 15 years. You know, I've been trying to solve the problem as early as possible and to avoid bugs making it all the way to production, with tests and static analysis and QA and all the things you can do before production." And that's a very valid argument. I think shifting left is something you should do. And so the question they u- usually ask me is, "Why should I shift right and just handle this stuff directly in the production environment? Isn't that going to create a culture where you don't really try to shift left because you can always fix it directly in production?" And I think that's a false dichotomy. That's a false… You know, it doesn't necessarily mean you either do the shift left or you have a proper runtime control in production. I think you need both because one is a safety mechanism to make sure you have the smallest number of issues in production, while the other is a emergency mechanism that make sure that when the bug or the something wrong makes it to production, you have a quick way to fix it instead of, waiting hours and hours, if not days. So you can think of it as shifting left is your prevention mechanism and shifting right is your I don't know emergency box where what do I do when there is a problem in production. You still need both. You can't have one or the other. It's not either/or. It's an and, in my opinion.
Darin
00:47:02.115
You can find out more about Unleash at getunleash.io. As a reminder, FeatureOps can be found at featureops.io, and all of Alex's information will be down in the episode description. Alex, thanks for being on with us today