DOP 356: Warehouse Robots Are a Distributed System

#356: Fleet management means one thing to a DevOps engineer and something completely different to Tomas Kovacovsky. To Viktor it is a CD problem - a fleet of Kubernetes clusters he would rather not babysit. To Tomas it is hundreds of physical robots rolling around a warehouse, picking orders, dodging each other, and working very hard not to lose their connectivity.

Tomas is the CTO of Brightpick, where the robots are not the kind you yell at for bumping into a chair. They are three-meter-tall autonomous pickers - some telescoping up to six - that find their way using lidar, recognize items with neural networks, and make their own decisions the second the network drops. Here is the part that will feel oddly familiar: everything you already do to ship software shows up again in the physical world. Canary rollouts. Rollbacks to the last good config. Prometheus scraping every robot, Grafana for the fleet. Logs, metrics, traces. Split brain, when a robot and the server disagree about what just happened. Even a flaky robot - one that feels off with no error to point at - gets diagnosed the same way you would hunt a flaky test: compare it against the rest of the population and find the outlier. A warehouse full of robots, running like a distributed system.

The stack is what you would guess and also not. C++ on the robots for speed, Python on the backend, Kubernetes on an edge server inside the warehouse because latency matters down to the millisecond, and Git as the source of truth - the on-site servers check for differences and update themselves. GitOps, for robots.

Then it gets bigger. The optimal pick speed, Tomas says, is infinity - right up until you try to pick an egg. The real bottleneck was never the picking, it was the traveling, so Brightpick moves the picking into the aisles instead of hauling totes back to a station. He also drops a prediction worth chewing on: the intelligence arrives before the dexterity. Machines will think their way around a warehouse long before they can fish for keys in a bag the way your hand does without looking. And the jobs question everyone braces for - the robot guys walking in, are you fearful for your job in 20 minutes - turns out the picker positions were mostly empty to begin with. Hundreds of thousands of them, unfilled.

The takeaway for anyone writing software is the one Tomas lands at the end. The craft is getting eaten. What is left, and what actually matters, is whether you can connect the work to the product.