DOP 333: The Hidden Problems Behind Every Data Pipeline

Episode 333

Show Notes

#333: Pete Hunt, CEO of Dagster and early React team member, explores the evolution from Facebook’s early React development through trust and safety infrastructure at Twitter, to building modern data orchestration tools. The conversation reveals how similar infrastructure problems plague every industry - whether you’re launching rockets or managing porta-potties, the core challenges remain consistent: late data, quality issues, and mysterious errors that require both automated solutions and human oversight.

The discussion dives into the technical realities of scaling systems, from the microservices complexity trap to the current AI adoption wave. Hunt shares candid insights about leadership challenges, including how well-intentioned technology recommendations can backfire, and why most data projects fail despite sophisticated multi-agent orchestration. The conversation touches on career advancement pressures that drive unnecessary complexity and the importance of focusing on actual user adoption rather than technical sophistication.

This episode features Pete Hunt in conversation with hosts Darin and Viktor, covering everything from regular expression nightmares to the future of data infrastructure and the lessons learned from building products that people actually use.

Frequently Asked Questions

What problems do all data pipelines have in common?

Pete Hunt of Dagster reduces it to three on DevOps Paradox episode 333: “My data is late.” The data does not look right by some definition. And “I’m getting weird errors and I don’t know why.” The industry names them data quality, observability and data downtime. He says the classes of problem are identical whether a company is testing rocket engines or delivering portable toilets, even though the data sets are nothing alike.

Why do data projects fail even when the pipeline works?

Pete Hunt tells Viktor Farcic on DevOps Paradox episode 333 that a stakeholder makes an urgent request, the data team scrambles and builds a dashboard, and then nobody uses it. He treats this as an adoption problem rather than a technical one, closer to getting people to engage with a consumer app. Farcic recognises the same pattern in developer platforms built on assumptions about what developers need.

What is Dagster?

Pete Hunt describes Dagster on DevOps Paradox episode 333 as open source data orchestration, with a commercial product on top. His comparison is Kubernetes: you declare what you want and a reconciliation loop keeps the actual state matching it, except the domain is data rather than pods, so it checks that data is hitting its service level agreements and flowing through the pipes. The underlying structure is a directed acyclic graph of data assets.

Why did so many companies adopt microservices?

Pete Hunt gives an incentives answer on DevOps Paradox episode 333: demonstrating large-scale distributed system architecture is written into promotion guidelines at big tech companies, so engineers build services. Vendors then fund tooling for the complexity that follows. His view is that “the day you introduce a new service should be a very sad day”, where at most large companies it is a happy one, and that company tech blogs were content marketing rather than technical advice.

How do you catch a spam attack that started five minutes ago?

Pete Hunt explains on DevOps Paradox episode 333 that trained models lag, because an adversary adapts faster than labels arrive and retraining happens. For emerging threats his team broke posts into trigrams and skip-grams and watched for sudden jumps in usage, which could be a spam campaign or a celebrity news event. Human reviewers triaged the top movers, a quick rule stopped the bleeding, and those rules then generated labels feeding the models.

How should you measure an engineer's performance?

Pete Hunt argues on DevOps Paradox episode 333 that the closest thing to a universal measure is forecasting ability rather than output. A mid or late career engineer working in a well understood area should be able to give a date, or give a range and then narrow it. Expectations scale with level: new graduates are expected to be wildly wrong. He adds that tying the measure to compensation ruins it.

What is the DevOps Paradox podcast?

DevOps Paradox is a weekly podcast co-hosted by Darin Pope and Viktor Farcic, covering DevOps, platform engineering, and modern software delivery. Episode 333 brings in Pete Hunt, an early member of the team behind React and now leading Dagster, to discuss data pipeline failures, why dashboards go unused, and the incentives that produced the microservices era. Every episode page carries the audio, the video, and a full transcript.

Episode Transcript

Read the Full Transcript

Topics

Share and Download

Guests

Pete Hunt

Pete Hunt

Hosts

Darin Pope

Darin Pope

Viktor Farcic

Viktor Farcic

Viktor Farcic is a member of the Google Developer Experts and Docker Captains groups, and published author.

His big passions are DevOps, Containers, Kubernetes, Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).

He often speaks at community gatherings and conferences.

He has published DevOps Paradox and Test-Driven Java Development.

His random thoughts and tutorials can be found in his blog The DevOps Toolkit.