Prefect just bought Dagster, another big Airflow rival — and it’s not a data pipeline story
The New Stack Paul Sawers
Prefect is buying Dagster, merging two of Apache Airflow's biggest rivals into one company. It's less about data pipelines now and more about building infrastructure for AI agents.
Based on reporting by The New Stack, Paul Sawers — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Seven years ago, Prefect and Dagster set out to do the same thing from different angles: give data teams a better alternative to Apache Airflow. Prefect chased simplicity and reliability for Python developers. Dagster leaned into defining what a pipeline should actually produce, not just the steps it runs through. Now Jeremiah Lowin, Prefect's founder and CEO, is buying his old rival outright, and roughly 40 people from the Dagster team are joining Prefect as part of the deal. Dagster and Dagster+ keep their names, pricing, and roadmaps, at least for now.
What makes this interesting isn't the consolidation of two orchestration tools. It's the reason Lowin gives for doing it. He frames the acquisition as assembling the pieces an AI agent needs to operate without constant hand-holding: Dagster to define and track goals, Prefect to actually run the improvised work, and FastMCP — Prefect's own connector framework — to control what an agent is allowed to touch along the way. Lowin's language is unambiguous about ambition, calling the deal a shift in the
My take — AI-written commentary, not fact-checked reporting
This reads less like a data-tooling merger and more like Prefect buying market share and mindshare before the AI-agent orchestration land grab really kicks off, and I don't blame them. FastMCP landing as Anthropic's official Python SDK the same month MCP shipped is the real story here, not Dagster's YAML components. Schrock's exit, dressed up as an advisory role in the press release but read as a clean break on his own blog, tells you who actually won this negotiation.
Read more about this at: The New Stack
Related stories
OpenAI and Datadog leaders back AI deployment startup Arrakis
Tech.eu · 2 months ago ·
21