Avery
00:00:00.000
I think the computer industry kind of lost its way at some point because we started believing that it's better and easier to constantly shift the platform underneath us and then constantly run to keep up with the constantly shifting platform underneath us. Rather than keep the platform stable, stop breaking stuff in the platform. You can keep improving the platform, but you can't break the stuff that depends on the platform.
Darin
00:00:25.396
This is DevOps paradox, episode number 3 1 6. Bringing back the original internet vision using tail scale. Victor, how many VPN clients do you have running on your machine today?
Viktor
00:01:41.994
Never again. So, dear company, if you want to send me an offer to join you, if you're using VPNs, I can tell you in advance, no.
Darin
00:01:51.329
Well, for the 99. 9999 percent of the rest of the world, you will have to deal with some sort of connectivity to your company infrastructure.
Darin
00:02:03.358
Well, okay. V I did say VPN, let's say VPN connectivity on today's show. We have Avery Penner run on from tail scale. Avery, how you doing
Darin
00:02:14.948
very well. Thank you. So VPN versus what tail scale does now. Tail scale is not a VPN, but at the end of the day, it's a VPN. Is that a, good, bad way of saying it?
Avery
00:02:31.028
Yeah, I like to think of it as It is a VPN in the technical sense. It exists at layer 3 so. It adds an extra layer 3 or 4 in the middle. It's a connectivity tool that operates like way, way down below the surface where you might normally be used to things happening. But it doesn't feel like a VPN and it doesn't exactly get used the way a typical VPN would get used.
Darin
00:02:52.852
Explain why that is, because, so explain what a typical VPN would be and then explain what a tail scale, now there's other, what I would call competition in your space. I won't name the names, but they do things differently than just what we're used to from a VPN from the eighties.
Avery
00:03:10.582
first of all, there's two kinds of VPN. and so if you're using neither kind, or if you've sworn off VPNs, it might be because you swore off just one kind rather than the other kind. But the original kind of VPN, which I think actually started in the 90s or so, stands for Virtual Private Network. So the idea is, it's the same as your old private network, which was a physical set of wires in your office. But it, is a remote access to that. private network that you had before. So you create, using encryption, this idea of a virtual network where you can put your private stuff, and it's private. the kind of VPN that almost everybody thinks of today, when they think of the term VPN, is actually not much like that at all. It is a service that you buy that routes your internet traffic out to the public internet. through a private link, So there's a temporary private link, and then it like, pops out the other end, and it's, it's what I might call more of a virtual public network. still an abbreviation of VPN, but it's a completely different thing. those are honestly like, interesting business models, for the most part, because they depend on you sort of having fear of your local network. And trusting that their network is less scary. and that's a useful thing sometimes if you're in a café Wi Fi, for example, and you're not sure if the café Wi Fi is encrypted or, or they have a, funny routing or going to be inspecting your traffic or interfering with your traffic. but it's, not at all the same as accessing your private stuff from a remote location.
Darin
00:04:30.990
You said nineties, I said eighties, because to me, it all started even pre eighties to banks of modems, to me what I see tail scale doing. It's very similar to what the banks and modems did back then as compared to a
Avery
00:04:46.299
yeah, that's probably pretty accurate. I think the original, I actually did some research on this at some point, so I would sound smarter when I talk about this. I had to look it up in Wikipedia. I think the first VPNs, per se, were created in the 1990s. It's interesting, I think maybe the first firewalls were created around the early 90s. which is interesting, right? So when you think about, like, banks with modems, I think that's what people called private networks, essentially. Like, you had a phone line, and you dialed into this phone line, and that was your network. Or you even didn't even buy a regular phone line, maybe you had a dedicated line from the phone company that only you could use, and that was your private network. But effectively, a VPN should make your computer feel like you're on a private network, sort of like back in the 80s or 90s. When people had like physical networks, or like even when you're at home you have a physical network. But it should make you feel like you're on that physical network, even when you're not physically on that physical network right now. And that's what Tailscale tries to get to. This feeling of like, hey, I have a private network that's just mine, nobody else can get on it, and I can access it and do whatever I want without worrying about the stuff that I would have to worry about on a public network.
Avery
00:06:01.461
is essentially exactly what a VPN is, right? It's a set of pipes that you actually run inside the regular pipes of the internet. And then you can do whatever you want with this VPN technology. And so one thing you can do is buy a consumer VPN and route your traffic, your public traffic, out through their pipes, which went through somebody else's pipes, and just sort of like, Double route it in the hope of maintaining some kind of privacy or security. And the other thing you can do is route back to your private stuff. And so the simple explanation of what Tailscale does is you can put it on any of your devices, or all of your devices, and then those devices appear in a virtual network that's only visible to you. And it doesn't matter if they're behind firewalls, or multiple levels of firewalls, or CGNAT, or dynamic DNS, or IPv4, or IPv6, or any of those things. Like, all of that stuff just sort of disappears. And you have your own IP address and your own DNS name, and you can access your devices from any of your other devices from anywhere.
Darin
00:06:59.883
but I, but I already know that the failings to my argument, but I'm going to state the argument anyway. So tailscale is a network stack that's running on top of my existing network stack.
Darin
00:07:11.474
That's got to be able to fall apart because I got software running on top of software. could go wrong?
Avery
00:07:17.767
what's interesting, the thing I like about, of course I'm biased, but the thing I like about the TailScale architecture is we actually didn't put the network stack on top of your network stack per se, we actually inserted it in the middle. which is kind of a funny distinction to draw. But what the world has evolved into, is this ever heightening network stack. Because what happened is the original internet was supposed to let you connect any device to any device, That was the point. When I got on the internet in the early 90s, and I dialed into a modem at the local university when I was in high school, I could access any of the computers on the internet just by like Telnet or FTP. And it was really neat. The internet handled that part for us. And then, over time, we started running out of IP addresses. We started noticing that like malware would get through. so we introduced firewalls. So it became like more unidirectional. And so now if you look at the way the internet is used, you don't actually connect one device to another device. You connect your device. to the cloud, that's actually kind of a throwback to the way it was in the old, old days when everybody had a dumb terminal that was connected to an IBM mainframe. that, you know, worked really well. But the world changed dramatically when we switched to essentially the peer to peer what we call sneaker net of PCs where Microsoft Build Windows boxes, and everybody had a Windows box, and you could run whatever you wanted on your Windows box without paying a tax to IBM, So we're back now to this world where like I've got a supercomputer in my pocket, But my supercomputer is useless if it's not connected to this big mainframe in the cloud that we call AWS. And that is an interesting model, but it's not the only model, but it's all built up because of the series of mistakes that has happened over the past 30 years as we were trying to build the internet. Like, it's no longer possible to connect even my phone to my laptop when they're sitting side by side because the infrastructure doesn't exist. What tailscale does is it adds like one little layer in, you know, the infamous OSI stack down at the internet layer that actually makes the internet do what the internet was supposed to do. Which is what it used to do back in the early 90s, but eventually we got lost, it's kind of weird it actually strips away all these like layers and layers and layers of workarounds we've added on top for the fact that the internet doesn't work and it just makes the internet work and Once you've sort of like twisted your brain the right way all of a sudden You can start to think about solving problems in this way that actually removes a whole bunch of the crud that you've added over decades. I realize that sounds a little abstract
Darin
00:09:50.791
Well, no, you, stated it very well. All the crud that we've added, that's windows. and to an extent we can
Avery
00:09:56.196
Well, there's so much CRUD that we've added, but also Windows, yes. I'm a Linux person myself.
Avery
00:10:06.621
I'm a Debian person, I realize, a little bit of a throwback because Debian hasn't changed since the early 90s, but that's one of the things I like about it.
Darin
00:10:15.066
So if you were telling somebody to start on Debian today, what would you tell them to start with? Would you tell them to start with the latest? Would you tell them to step back one?
Avery
00:10:22.780
the nice thing about Debian is like even the latest is like stepping back one. Uh, so you're always pretty safe, downloading the latest Debian stable. and it's probably three years older than the latest Ubuntu.
Avery
00:10:35.853
Ubuntu, they, change stuff, and not always for the better. They're always like moving things for the sake of moving things. And I, that's one of the things I haven't really liked about them.
Darin
00:10:44.710
let's stay there for just a second. Not necessarily bashing on Ubuntu. Cause that's, that's not the key part of this, but Debbie into Ubuntu. When you go into larger corporate organizations and they allow the use of Linux, right. In this, this year of the Linux desktop, that will never arrive. A lot of times they won't allow anything that doesn't have quote unquote support.
Darin
00:11:14.310
You are required to run Ubuntu instead of running Debian, which is true, or maybe running Fedora. Now we'll go there. I prefer the Fedora line. I'll just put it that way. All right. Just the variants of that, but that's, that's me.
Viktor
00:11:26.700
Most companies are still trying to figure out how to allow Mac over Windows, Fedora, that's such a stretch, but go on, go on, dream.
Darin
00:11:37.775
exactly. But my, real point here is if companies are. Forcing specific operating systems. We'll say it that way from my perspective, from a, what I would think is us tail scale, it's like, well, rats, we have to create how many different distributions of tail scale and how many different stacks, right? What, what does that look like? I mean, how, how do you even, I mean, to me, the Cartesian of that potentially is just a numerable innumerable. I don't know.
Avery
00:12:09.617
We don't even know the word for how hard to count it is. don't remember the exact number of builds of Tailscale we have. I think it's something in the order of like 89, per version. And it's because every single Linux distribution does things slightly differently. and there's multiple versions of Windows. There's actually three, I think, different ways to deploy Tailscale on macOS, depending on what your corporate requirements are, or your personal preferences are, including one is in the App Store. Uh, there's an iOS app, there's an Android app, there's one for Apple TV, there's one for Amazon Firestick. There's one for Raspberry Pi. There's a, actually there's a standalone build for Raspberry Pi. I don't know if you've heard about it. it's called Go Crazy with a K. instead of the Linux user space, it just uses the Linux kernel plus one giant Go app. And they've built Tailscale into the one giant Go app, and that can run in your Raspberry Pi, and it can be a little router. There's an OpenWrt build. There's builds that run on people's drones. There's ones that run on little embedded systems. self driving cars. Uh, we support all of it. And I'm just, I'll add one more layer of crazy on top, is TailScale supports every old version of TailScale that has ever been released. And it can talk to every new version of TailScale that has ever come out. and they can all talk to our TailScale control plane in the sky that coordinates, the different versions talking to each other. And we made that promise from the very beginning of the company because one of the things, well, first of all, as I mentioned, I don't really like it when things move around for no reason. But also like The network needs to keep networking, If you need to keep updating your devices in order for them to be able to access each other on the on the network then you've got a problem, because not all of these devices are going to be turned on. Sometimes they get turned off for three years and only powered on in an emergency, and when that emergency happens, you need that device to still work, So we put a lot of work into this kind of compatibility. Tailscale runs on just about everything. It's portable to anything that it hasn't already been ported to, and it will talk to every other Tailscale that has ever been made.
Viktor
00:14:15.151
I mean, I understand the benefits, kind of, of course, if you ask me, should this run on anything, anytime, and so on and so forth, I would say yes, if there is no cost. But, you know, like you, you said Apple TV, how many Apple TVs have 10 scale?
Avery
00:14:33.248
quite a few, actually. We made a video about Tailscale on Apple TV. I think it was our most popular ever, Tailscale video because people instantly connected to it. It's like, oh, I wanted to have a device at home that I could use to relay traffic to the other devices that I have at home. And I wanted it to be always on, but I didn't want to have to set up a Raspberry Pi because, like, Linux stuff is too difficult for me. So if you go into an Apple TV and you download Tailscale from the App Store, it can relay your traffic. from the outside world onto everything else on your local network, which is fun. It can also connect your Apple TV to different local networks that you might have devices on. So if you carry an Apple TV with you when you go to hotels, because you're tired of the smart TVs that they have in hotels, your Apple TV can get back to all of your devices that you have at home, like Plex or something like that, and watch your videos that you have stored on local storage, for example. you can also use your Apple TV at home as a so called exit node. I talked about the two kinds of VPNs before. You legitimately might have the need for those two kinds of VPNs. If you want to, if you don't trust the traffic in your local cafe or in the hotel that you're in, you might want to route your traffic back home. through your Apple TV so that it emerges from your home internet. and this has been useful to me, for example, when I'm traveling in other countries and for some reason I'm in Canada, my bank here won't let me log in. I think it probably is just a bug in the banking software as opposed to a security feature, but it won't let me log in when I'm not in Canada. But if I tunnel my traffic back to my Apple TV at home, it can emerge from my Apple TV and look like I'm coming from Canada and then I can use my bank. so Apple TV is actually a really popular use case. Uh, to get back to your question about how hard it is to test all this stuff, it's pretty hard, but I think as, uh, you know, this is me, maybe I'm going too philosophical on this, but, I think the computer industry kind of lost its way at some point because we started believing that it's better and easier to constantly shift the platform underneath us and then constantly run to keep up with the constantly shifting platform underneath us. Rather than keep the platform stable, stop breaking stuff in the platform. You can keep improving the platform, but you can't break the stuff that depends on the platform. If every level of the platform that you're built on absolutely promises to not break stuff, then every new layer you build is a lot easier to make not break the next layer, It all depends on the whole system. having this cooperative set of commitments to not break the other stuff. And the whole world just sort of fell apart, probably because we're so worried about security, We use security as this excuse to put everybody on a treadmill because you can't not patch your code, because what if the code has a security hole? And if there's a security hole, I have to get the latest version, and the latest version breaks stuff, and like, too bad, everything else now has to break. But that's okay, because you have to upgrade that too, because of security holes. And like, what if you just don't have security holes? What if the platform, when you fix the security hole, doesn't break a bunch of stuff? It really changes all your assumptions that you didn't realize you had about how software needs to be built.
Darin
00:17:36.446
so what if your code has a security hole, uh, reality check, your code has a security hole, whether it exists today or not, there is one.
Darin
00:17:48.689
How do you deal with that? Because if we're concerned about our network being stable, get down another rabbit trail real quick. Uh, the reason why everything keeps changing is because somebody needs to make their bonuses this month.
Darin
00:18:01.897
I don't want to say it that way. Okay. I'll come back to reality now. How does tail scale deal with that security thing? Because I mean, as soon as you write a line of code, there's gonna be a security hole.
Darin
00:18:14.514
I'm, I'm being extreme here, but how do you defensively deal with that? Especially if you've got a say, look, I watched a movie the other night that was based on Brit van Wink. If you don't understand that story, go Google it. It feels like you guys went forward in time and back in time to get to the point where he's like, this will always work
Avery
00:18:37.709
We're kind of like time travelers. We went back in time and brought stuff into the future. Um, it's not as hard as doing it the other way around. so, when I was in high school, way back in the 1990s, I worked at a computer store. and I made a Microsoft Access database for managing, like, our customer list. as anybody who's used Microsoft Access knows, the thing is just like a walking, talking security hole. Right? It is not possible to build a Microsoft Access database that is not a security hole. The whole way it works is you give somebody read write access to a big file that's the database, and you put it in a file server, and then everybody in the office accesses this one big database file from a, Windows client app that pokes at the file and hopefully doesn't corrupt it. And of course it does sometimes corrupt, and then you restore it from backups. why didn't it matter? that my computer store inventory and customer tracking system had a million security holes in it. And the answer is, it wasn't on the internet. When you put stuff on the internet, all the people who are going to exploit the security holes can find it. When you don't put it on the internet, when you don't even have the internet, then the only people who are going to try to exploit your customer database slash inventory system are the people who work for you. who have access to it because they are literally standing in the store right now. for something as low valued as an inventory system, the probability of one of your employees going through and trying to like, corrupt the inventory is extremely low. Or steal the information about your customer data, which they already have access to, because those customers are walking in and out of the store all day. You have to understand your threat model in order to understand what security holes really matter. Exactly because everything always has a security hole. Everything is a security hole, right? Like, I can walk into a store, and then walk out with some stuff, and not pay for it, and they might not catch me. In fact, there's lots of ways to do that without getting caught, That's a security hole. Why doesn't it matter? Well, it does matter. But people just sort of work it in as a percentage of their inventory, right? It's like, oh, there's going to be, I think, what do they call it, shrinkage. but the nice thing about that shrinkage is it almost never escalates to the point where it's like, this is a massive problem, Because most of the time somebody doesn't, go into the store, smash the glass in the middle of the night and walk out with everything. And when they do, we have other systems on top to catch those things. In software, we've decided that the Internet is the model, Everything now, every computer all the time, is on the Internet. And so, if I'm going to run a service that people need to access, it needs to be on the Internet. And if it's on the Internet, then billions of people have a chance to try to attack the software that I'm using. And when you have billions of people, even if like 0. 00001 percent of them are the really bad kind who are going to find a way to exploit your random app for low value, Those people exist and are going to do it, If you can find a way to make it so your app is just not on the internet, because it never needed to be on the internet, because it was only supposed to be shared with like 10 people, then all of a sudden The whole equation is different, Those ten people have a very, very, very, very low probability of being the kind of people who are going to invent novel new attacks to break into your specific app and do something with the data that actually has negative consequences. So the real trick is to build a system that actually can make you feel safe. It's a lot like human social networks, Where, if you trust a billion people, you're gonna get in trouble. If you trust ten people, It's not always going to be perfect, but the kind of trouble you're going to get into is a lot less.
Viktor
00:22:11.149
How do you choose those 10 people? Because that's one of the problems that I see in many newer companies trying to make it into the SaaS area, right? That other companies say, you know what? Your product is going to be self managed by me. because I don't trust you.
Viktor
00:22:36.363
you would be AWS, I would run this happily, right? Because I trust those guys. They're the only guys that I trust. And I'm using AWS as example, right? So there is that selection also going within companies that, okay, so if I need to trust 10 people, let it be, you know, big guys, Oracle's and Microsoft's and Google's of the world, which basically prevents almost. anybody else entering into the game.
Avery
00:23:07.390
it interesting that society has come so far that we now think that trust should only be given to the biggest entities with the most money? I would say that that is a really surprising outcome. because historically I think we probably would not have trusted the biggest entities with the most money. There's certain kinds of things you can trust big entities to do. But you probably can't trust them to have your best interests at heart, We've built a whole system where, like, even though these big entities don't have your best interests, they have a reputation, so, like, maybe they can't, like, too overtly, pick your pockets. Or steal your data and sell it to other people or whatever. They have to kind of keep it, like, sort of on the level. But, historically, humans evolved to trust not big, giant groups. They evolved to trust the people that are close to them. Because the people that are close to them are tied into a social fabric with them. So the people you naturally would trust the most are your family, your extended family, your friends. Maybe your village. And you're kind of like, maxing out. There's this sequence of numbers. They're called Dunbar's numbers. You've probably heard of the famous Dunbar's number. It's around 200 or 150. Somewhere in there. If a company grows beyond that size, people can't remember each other's names anymore. so the mental aid for that one is like, around 200 people is when you start needing name badges at your company. because you can't actually tell if that person works here or not, because you don't know them all. But actually, there's some research that Dunbar and some of his associates did, back in the day, that basically says every three to five times multiple of a social group size, the dynamic changes, And so to get to that 150, 180, 200, we've actually gone through several different Dunbar's numbers and they have the parallels of like, family, household, extended family, village, and so on. the smallest sizes, you can choose to trust people, and you can choose to, eject people who can't be trusted. And then it's as simple as that, If I have to buy SaaS products from a 20 person company that's outside my social network, then of course I shouldn't trust them, because who are these people, They don't necessarily have a reputation, so the reputation based system doesn't work, and they're not in my social network, so the social network system doesn't work. It's kind of the, like, uncanny valley in the middle. But if I make my own app, and I let My fellow employees use my own app. Well, the trust network there is perfectly fine. And so the world we've gotten into is one where we forgot that you can hire a high school student at your computer store and make an inventory system. And that inventory system is not going to leak all of your personal information to, I don't know, foreign governments. Instead, we've moved into a world where everybody can make an app and try to sell it to everybody else, but has no reputation whatsoever. So instead of trusting them, we have to trust these mega corporations who are like guaranteed to not exactly be trustworthy, but we try to keep them in line with this more complicated system. But it feels, unnatural, When you actually start to think about it, it's really awkward that I can only trust like Microsoft, Google, and Amazon. that's, not how I want to be, Nobody wants to be like that. And we all know we can't trust Microsoft, Google, and Amazon.
Viktor
00:26:28.444
that's true, but we trust them precisely for the reasons you just mentioned, right? It's almost evolutionary, right? I would summarize what you just said as, I trust people I know, right? It's just as likely that my uncle is a criminal as any other random person, right? Statistically, it's just as likely. I trust my uncle because I know him rather than anything else. Not because he's a better or worse person, right? then people would apply that logic to Microsoft, let's say. Not because necessarily Microsoft is big, because I know what is Microsoft. I heard about them, right? I don't know what is Acme. Or whichever other company kind of, I don't know, this is the first time I hear about them, right? So it's familiarity rather than anything else. And I completely agree that there is no valid, logical reason why I would trust Microsoft over Acme. But evolution tells me that I'm wrong. Kind of like, most of human history, and this is my interpretation of what you said. actually tells me that I should trust those I know, rather than those that are validated to be
Avery
00:27:48.296
Right, and I think the connection that we have to remember, so what you said is true. Statistically, your uncle is no more likely or, no more or less likely to be a criminal than any other random person. The difference with your uncle is, by now, you have a pretty good idea of whether he's a criminal. Right? And so you know exactly what level of trust to give him, and what things you can trust him with, and whether he's loyal to you, and is going to try to rob you, or rob other people, And that completely changes your ability to act, Because you know enough about the people in your close social network that you can just like, set up your trust boundaries, and you know, grant permissions to them if you want, and let them do things or not do things. based on how much you know for sure you can trust them. As soon as you get outside your social network, you don't have that ability anymore. Everything has to be this unnatural layer of like, well maybe the government's gonna protect me if these people are too bad. Or maybe like an insurance claim will protect me if all of my stuff gets stolen. Or, etc. And this is what makes all of interaction on the internet feel so weird to us, because when you get out of your actual social network, as opposed to these computerized social networks that don't work like human social networks anymore, nothing works the way your body or your mind is intended to work. And that stuff has evolved over thousands of years, and actually works really well. So we've kind of gone down this super weird path, and just the last like 30 years or so, and most of social networking is only like 20 years old, Where we're doing things that are not natural, and there's no particular reason to believe that the thing that we invented in the last 20 to 30 years is better than the thing that humans evolved into over the last thousands of years. And so this is where I can tie it back to tailscale. It feels really weird. The name tailscale is the hint, right? It's the opposite of It's all about the long tail of small things and small groups and a very large number of small groups, Tailscale is about going back to this idea that trust can work really well in small groups without doing anything fancy, right? If you can just produce a set of computers or services or whatever that are only accessible to the people they should be accessible to. If you can trust that one thing, that nobody can access my thing. Unless they're actually supposed to, Imagine a world where you can actually believe that to be the case, as opposed to believing that there's probably hundreds of security holes that are going to make it possible for any, any idiot anywhere to be able to access it. Only a dozen people who are supposed to be able to access this dashboard or this database can access it. If you can do that, then all of the rest of your work is easier. like suddenly you don't have to be paranoid about every single thing in this database, every single thing in this app. You can go back to using Microsoft Access type stuff because you know that nobody's going to get access to this Microsoft Access who's not supposed to. so it's really just like tying. thousands of years of social network evolution back into computing and say like, all of that was broken, let's do it the way we're actually going to do it that feels natural. And the experience that people have when they use Tailscale, networking people, or like computer nerds anyway, they install Tailscale, and they're just like, whoa, why does this feel right? I can't even explain why this feels right, but everything about this starts to feel right. I can access my computer wherever it is, I can send a file from my laptop to my phone. even though they're not made by the same manufacturer, that's actually a thing that's really hard to do. it doesn't have to go to the cloud and back. It can actually go over my local Wi Fi because the two things are side by side and they just go over the local Wi Fi. But if they're not on the same local Wi Fi, if I'm in a cafe on my phone and I want to get back to my laptop that I left at home, it has just as direct a connection and just as trustworthy a connection and it doesn't go through the cloud. It just connects the two things because they trust each other. And so just everything starts All of the, this complexity that we've built up over 30 years that we just got used to, that they've talked us into believing has to always be there, it just goes away. And it's very hard to explain what that feels like without actually feeling it, because most of us have never in our life.
Viktor
00:31:48.739
To me, that makes perfect sense. So, let's say that I trust myself, I trust Darin, and I want to connect to Darin. Uh, and if I can do it directly, or as directly as possible, the better, right? Perfect sense.
Viktor
00:32:03.039
But then, we trust each other. But the only thing that, actually, the only foreign thing here, between me and him, that I might potentially not trust, would be Telscale, then. And I'm playing devil's advocate here, right? But, how do I trust you guys? Right? You're the unknown entity here.
Avery
00:32:23.598
Well, absolutely, right? And of course, we get that question a lot, because like, do you do with trust? Generally speaking, and, you know, I think we talk about zero trust networking, It's one of the popular trends nowadays. And what do you mean by zero trust? Like, you can't actually have zero trust. What I like to say is, like, nobody actually wants zero trust. What they want is trust, But where can you get it? Like, trust has to be earned. trust can be bootstrapped from one thing to the next. If I trust you, and you say, hey, this person is really trustworthy, then maybe I don't trust this person that much immediately, but I trust them a lot more than a random person, again, that helps you get past this question of, you know, if everyone in the world has an equal probability of being a criminal, then how do I know? It's like, well, the person that I'm pretty sure is not a criminal, who just introduced me to this other person, and then they say, I'm pretty sure this person is not a criminal, that person has definitely a lower probability of being a criminal than a randomly selected person, And so, tail scale. One answer is that if you trust Tailscale, you can stop trusting so many other people. So it's a net improvement. That's not that much of a, I don't know, for me, who's a security person, I don't know if that's that compelling, but it is a statistical fact that if you can get away with trusting fewer people, you'll probably have less problems. The other answer is we have this feature called TailNet Lock, that, basically makes it so that no device can join your network unless you have signed the keys belonging to that device. So I guess I should go back one level. So, TailScale is an encrypted VPN. Every node generates its own public and private key pair. It only shares the public keys with TailScale, and TailScale sends those keys down to your other devices, and says, here's the list of devices that you personally have authenticated, into the Tailscale network. The private keys never leave your device. And so we never have them, which means we can never decrypt your traffic. So the only threat that Tailscale can have to you is that we can let people into your network that shouldn't be there by lying and saying, here's some public keys, that should be on your network that you have in other locations said that you trust, And then it's not actually true. We've actually inserted our own device and then it can have conversations with your devices. we still can't decrypt. the conversations that you want to have, but we can create new conversations. And so this tailnet lock feature says like, okay, I'm not gonna you don't have to trust public keys that come in through the tailscale network unless you've got some other thing on the side, like basically signing them however you want to sign them. Now that adds a lot of complexity. Most of our users don't do it. But it is a thing you can do that basically eliminates the first of all, the tailscale hole, like trusting tailscale is no longer necessary. And actually, one layer up from that, the way Tailscale works is you download the app from the App Store, you log in with like, we don't do usernames and passwords ourselves at Tailscale, we always use some kind of SSO. So you can log in with your own OIDC, you can log in with Google, with Microsoft, with whatever. And that's the bootstrap identity, it's like, okay, everybody that's appanoir. gmail. com, which is my, personal email address. If I log in as appanoir. gmail. com using Login with Google, every device like that can connect to every other device like that. Now, an identity provider can also have security holes, right? And we know that they do sometimes, right? If you use tailnet lock, even a security hole in the identity provider doesn't reflect all the way down into your network. So we have all these different things that we put together in tailscale to try to make it so it's like, okay, do you need to trust tailscale? No. Do you need to trust your identity provider? It would be nice to trust them a little bit, but if you're really paranoid, you don't have to trust them at all. Right? And that eliminates all the different places where the threats can come in. And if we do the work of eliminating the different places where the threats can come in, then you can get back to work. The thing you're trying to build doesn't have to deal with any of that stuff. Every connection coming into a device through the Tailscale network already has an identity attached to it, so you don't even need a login screen. And so you can't screw up implementing the login screen. You can't screw up implementing the authorization stuff because someone who shouldn't be able to access the service never saw it. It doesn't exist to them. because you can't screw it up, that's one less security hole that you can introduce into your app that matters.
Viktor
00:36:32.655
It feels like, you know, when you want to start investing your money, and it very quickly becomes scary. Oh, I don't know where to invest, I don't know, should I diversify, should I do this, should I do that? It's just scary because your money can easily disappear, right? But then comes the guy. It says, if I will give you all those guarantees, if you invest everything through me, you just give me your access to the bank account, I will take your money, I will invest, I will keep the record of everything, you can see what's going on at every given moment. that's even scarier in a way, right? Because then you ultimately trust that guy. I'm not saying you're that guy, just to be clear, but it's how you very often how people actually lose money, right? Because they trusted the guy.
Avery
00:37:32.616
Yeah, absolutely. I mean, I think I realized when I talk about tail scale, and I haven't even said yet that you can set the whole thing up in five minutes, right, which people are like, Okay, I get all this stuff. And it's only five minutes because it's so easy. It's like, well, it is. And it all sounds too good to be true, right? And it sounds too good to be true, because we reduced all these layers of mass, and we just like cut it away and said, like, here's the place where all of your problems started. And why don't we just not do that? There's no reason to just trust me. I'm some guy on a podcast. We did open source our client software. So people who, have, and if you want to, you can. You can read exactly how the node software works. You can see exactly that the private keys never leave your node. There's an open source tailscale control plane called headscale. You can see exactly what goes on in the control plane in the sky. we have a white paper about how tailnet lock works, and you can also read the implementation of tailnet lock to be sure that it actually works that way. That's sort of the open source version of the guarantee, which, for most people, if we're honest, is not that valuable a guarantee, Because most of them are not going to actually read the source code, and most of them don't have the ability to read the source code, and even if I read the source code, I wouldn't be able to tell you where all the security holes are in the source code, But we have some faith that other people have reviewed the code, that it actually works the way it says it works.
Viktor
00:38:51.293
That's the, value of open source. I feel right that I, I'm not going through all the source code, but. others might have, right? But then, I'm playing a devil's advocate here, then that would give me a confidence. Hey, look, this is, open source, right? I haven't been through all the code, but it's open. Somebody might have, we would hear about, we would hear complaints, right? That there is something, right? Potentially, maybe, who knows? But that would incentivize me to go To do the extra mile and set it up myself again in AWS, right? Because it needs to run somewhere. but not necessarily to, to use Telscale as a service, right? Because the fact that I can see every line of code does not mean that you're running it
Avery
00:39:41.836
well the neat thing about tailscale is that you're still running the open source clients even if you're using the service in the cloud. And because of the way the protocol works, and of course you have to have somebody who you trust to analyze this protocol, but like, we actually can't do anything using our service in the cloud that's going to get you in trouble. Particularly if you use tailnet lock, which is just, you know, a key signing protocol implemented entirely at the client side. So you can see exactly what you're locking and exactly what you're trusting by this company up in the cloud. So you can choose whether, you know, the level of trust you want to have in us. And there is always this sort of like trade off between trust and convenience. And we let you sort of dial that trade off to wherever you want it to be, I want maximum convenience? Well, do it the easy way. Download Tailscale from the App Store on your phone and on your computer and then log in with Google, which means you have to trust Google, and you have to trust Tailscale, but then all this stuff happens. And it's like, okay, well now I'm, I'm ready, that was easy, I've done some stuff, but I want to, like, dial up my paranoia factor higher, then you can use this tailnet lock with the exact same configuration. And then we don't have the, you know, we and Google don't have the ability to mess with your network anymore. So whatever is going on in the cloud, you have, all it is, is a thing that distributes keys for you. Our cloud service is very simple. Yeah, I'm sure our engineering team that works on it would criticize that statement. But fundamentally, not a very complicated concept. And those keys that it distributes, if they have to be signed by you on a client device that you control, Then we don't have any ability to do something that messes with it. And I think that's a really elegant part of this. The whole protocol is like the removal of the need to trust anybody if you don't want, but your ability to choose who you do want to trust, hopefully a small number of people to make your life easier if you want to.
Darin
00:41:29.731
Why did you decide to open source the clients? It seems like, as a company, you would want to build a moat around that,
Avery
00:41:36.674
there's multiple reasons. I'd say, like, one of them is, like, I got my, career start in open source. Like, I would not be a programmer today if it weren't for so much open source stuff. I, I downloaded Linux floppies from BBSs, back in my modem days. Uh, and I learned all sorts of stuff from all sorts of things that way. so I, I have a real attachment to open source. I believe it is a good way. Uh, a more self serving reason is just that, people don't want to install proprietary software on their like open source server machines, So one of TailScale's primary use cases business is you put TailScale on like a cloud instance somewhere. You don't open up any firewall ports, and now people inside your company can access the services that are running on this cloud instance. If the only thing on that cloud instance that isn't open source, is tailscale, then people are really going to think twice about whether they want to, sort of like dilute their open source feelings, about the stuff installed on this device. And so we made sure that you don't have to make that hard choice, If your operating system is open source, then tailscale is open source. there is a, an element of giving away your moat, sort of, but, I guess I also have two thoughts about that. One is that this is a technology that should be everywhere. Just like the internet should be everywhere. Like, you know, we, you know, have a blog post called The New Internet. And it's like pretty, well, first of all, it's a funny joke because it's, you know, Silicon Valley, the TV show, talks about the new internet. but also, the internet only works if every Everybody uses the Internet, And so this needs to be everywhere. It probably is not going to be possible for one company to control the entire new Internet protocol, So we need the Internet protocol to be out there so that everybody can use it so that it can become a standard. So that, you know, my, the analogy I always use, 1994, Windows didn't have TCP IP in it, And then Windows 95 came out and included it, And now I have a watch. And if you buy a watch and it doesn't have internet on it, your watch is broken because it doesn't even tell the time, that's the level of ubiquity you need to be able to achieve. And to get there, you have to give up a little bit of control.
Darin
00:43:36.359
but giving up control. I don't want to give up control. It's like, I've been listening to everything that's been going back and forth, and the one thing that keeps coming back to me is least privilege.
Darin
00:43:49.394
Let's rewind back to the time when Al Gore created the internet. That's a political
Darin
00:43:55.224
Okay. What if instead of doing what happened? They instead overlaid least privilege from the day one scenario. Like we've been wedging in modems and VPNs and now tailscale to try to make us secure. What would have happened if they would have just implemented least privilege from day one?
Avery
00:44:21.429
I mean, to be fair, they just would have done it wrong. right? I mean, there's no way we could have known what the internet was going to grow into way back then. It was just impossible. If you look at IPSec, that's the closest analogy I can think of, they did it totally wrong. It's the wrongest wrong you can imagine. And they knew by then, for sure, that like, oh god, what were we thinking? Not encrypting every single thing on the internet, It's like, we've got to encrypt every single thing on the internet. And they still did it wrong, Just because we just didn't know enough about, like, at the time that IPsec came out, the reasons that it wouldn't work were not known to cryptography, And now we know a lot more things than we did back then. And so we have to, I mean, all of engineering is, is a matter of like, oops, well, we learned something there. I guess we should do things differently in the future. Now we've got all this legacy stuff. How are we going to do? The new stuff in a world where there's all this legacy stuff, we have to do that. But like now, encryption is like, you have to have an encrypted network. Identity is only like, you know, Tailscale is adding this to the network layer, but it hasn't been there ever. Even IPSec doesn't really have an identity concept to it, And then least privilege, you have to have first encryption. Second, identity. And third, how are we going to decide what your privileges should be? That was impossible to layer on until we got the other two parts right. So, TailScale, incidentally, also does least privilege. We have a really neat ACL system. Ah, but I don't need to do a TailScale feature list for you.
Darin
00:45:57.508
I could now put that silly access database on and not be concerned. Yes. I know it's full of holes, but I just don't care because I've least privileged it so hard that everything's going to be okay. I'm the only one and the boss is the only one. So there's two only ones, I guess.
Avery
00:46:18.708
Yeah, exactly. I mean, back in the day, of course, everybody just used a computer, like, a shared computer that nobody ever logged into. And so, at least Privilege was cool. Literally physically, physical access to this computer, A few years later, Windows NT like started forcing everybody to log in. which was like this, I remember a huge kerfuffle because people are like I don't want to have to log in, this is ridiculous. I can just, I want to go to my computer like I did before and have access to stuff. we've passed that point, now we have to log in. Now least privilege is
Darin
00:46:49.690
You actually bring up an interesting point there, shared computers still do exist today. And let's play out, let's say you and I share a computer, what happens with tail scale when I log in as me versus you logging in as you, because going back to, the tail net lock, like if we're going to, and we're putting in the ACLs, right, we've locked it all the way down to where only this person, this machine can get, whatever. How does that work with tail scale?
Avery
00:47:20.686
I can answer multiple layers for that. First of all, I have to start with the first layer, which is like, unfortunately, nobody has figured out how to make shared computers secure. because if you can do something on that computer, chances are you can find a admin escalation. Uh, it's sort of like a truism in security that if you have the ability to execute code on the computer, you can probably find a way to escalate to admin rights. that is among, you know, in the hierarchy of different security holes, that is the easiest one to get, is once you have remote, or code execution privileges, you can probably escalate to admin. If you can escalate to admin, then you can escalate back to the other user account, so any keys that that other user has access to, you fundamentally have access to. let's assume for a moment that we don't believe that's the case, and that it is okay to run a shared computer. and unless you are a really advanced attacker, most of the time that's the case. if you've got a relatively sort of okay level of trust between the different people, as opposed to randoms on the internet being able to access this computer and log in and have a shell account, then we won't worry about them doing advanced level escalation attacks. tailscale does work in that case. As long as only one of them is active at a time, because it does take over the network stack. And so, the network stack, when you are logged in, now makes all of the network access come through as you, If another person is logged in in the background, they might have access to some of your networking stuff, and you sort of have to choose whether you want to allow that or not. for example, Windows Terminal Servers, Usually what you'll do is put one instance of TailScale running as admin in the background that everybody shares and then the shared identity is just like this is a computer that's running Windows Terminal Server and so you have to set up your network rights using that. But that's, tends to be surprisingly okay with people because usually it's the Windows Terminal Server that is the service that people are accessing. It's usually not that they go into this multi tenant Windows Terminal Server and then want to go out to reach out to other things.
Darin
00:49:15.408
Sorry, I took us down a really deep dark hole there and you, you brought windows terminal server right into it and it's like, Oh,
Viktor
00:49:22.488
of, uh, uh, why, why Windows in a press, wouldn't we solve most of our problems if we would just ban Windows?
Avery
00:49:30.463
Yeah, sorry, forget everything I just said and let's talk about Linux machines that you SSH into. Uh, it's pretty much the same.
Darin
00:49:37.868
same thing. Yeah. So tail scale can be found at tail scale. com. That's T a I L scale. com. Just like you said earlier, internet scale, tail scale. That's a good way to remember it. All of Avery's contact information will be down in the episode description. Everything's being with us today.