ADR-25-02: Orchestration Tool

Status

Accepted

Impact

Low

Demand Owner

Imran Khan

ADR Contributors

Anish Kejariwal

Informed

Aaron Archer Waterman, Alex Weigel, Gautam Singh, Henri Dwyer, Hugo Lam, Merima Birkhimer, Mark McCreary, Muller Mu, Ramachandran Prasad, Vitek Trela, Yogesh Pandit

Publication Date

October 29, 2025

📗 Background

Today, in the CoE organization we lack a standardized, modern orchestration platform for data pipelines and MLOps orchestration. This gap creates fragmented approaches, higher operational overhead, and slower delivery to our stakeholders. Since 2024, parts of the legacy gCS organization has been leveraging open-source Dagster (OSS) in production with success:

  • AI4DD (formerly MLDD): first adopters of Dagster

  • gRED MLOps: as part of initiative approval with gCS LT in 2024, Dagster was selected over Kubeflow.

Additionally, the SADR product team (in legacy gCS) ran a side-by-side comparison of Airflow with Dagster for data pipelines and recommended Dagster, further advising that Dagster+ (enterprise version) should be used to scale more widely in the organization.

Based on these experiences, the gRED Data Ecosystem leadership decided in April 2025 to fund a 1-year Dagster+ license with a limited number of seats (15) as a seed-and-grow strategy to determine if this could be a preferred orchestration tool for gCS. This will cost ~$150k (inclusive of 12 seats, with additional seats available at $1200/seat pro-rated, and 3 admin seats at $800/seat). Procurement is expected to complete in early September.

ℹ️ Context

The decision to purchase Dagster+ was made in the legacy gCS organization as part of the gRED Data Ecosystem initiative. With the creation of the CS-CoE organization in July 2025, this ADR documents the rationale, scope, and strategy for Dagster+, and piloting it as a potential candidate for a CoE wide orchestration platform.

🤝 Outcome / Decision

We will complete the purchase of the 1-year Dagster+ license, which will still be funded by the 2025 gRED Data Ecosystem budget. This decision:

  • Does not establish Dagster+ as the data pipeline orchestration standard for the CoE, or the gRED Data Ecosystem

  • Does not imply Dagster+ will be the standard for MLOps in CoE (note: pRED MLOps currently uses Kubeflow)

By starting with a limited number of Dagster+ seats, this provides us with a seed-and-grow pilot strategy that aims to deliver value to current Dagster users in the organization and further evaluate Dagster+ for broader CoE adoption and as a potential standard.

Rationale

Dagster was selected by legacy gCS because it provides a unified platform for data and ML pipelines, and demonstrated key advantages compared to alternatives

  • Developer experience – asset-based approach, intuitive UI, and graphical representation of pipelines make it easy for developers to get started, and an opportunity for scientists to build data pipelines.

  • Momentum and adoption: success in production in gCS, and rapidly growing industry adoption and active open-source development, particularly in comparison to Kubeflow (registry, trainer, kserve, pipelines) in 2024 (Note: momentum should be re-evaluated as Kubeflow may have gained new momentum in 2025, as it is recently part of the CNCF).

  • Integration with key frameworks – native integration with dbt and OpenLineage for data engineering, and W&B for MLOps.

  • Observability – strong monitoring UI making it easy to spot and debug errors in complex workflows

  • Iteration speed – supports partial runs and resuming from failures, accelerating development cycles

Additional information can be found in the Resources section.

Alternatives Considered

  • Kubeflow: Successfully in production in legacy D&A, but it comes with a more complex API for pipeline definitions, which might be a hurdle for adoption on the gRED-side - knowing the lack of relevant experience and capabilities in the North-America based teams. Note: the pRED-MLOps team has been successful in abstracting this complexity in the Kubeflow Pipeline API largely away - by providing user- and self-service-friendly MLOps-SDK, reusable Kubeflow Pipeline components, and Bring Your Own Container (BYOC) approach. Kubeflow pipeline for both data pipelines and ML pipelines.

  • Apache Airflow: Used in legacy D&A and legacy gCS. Limitations in dbtCore integration, and weaker observability, traceability, and lineage compared to Dagster.

  • SageMaker Pipelines: Previously used in legacy gCS, but experienced very low, and slow pipeline deployment.

Since the evaluation was originally done, there are additional options to be considered for evaluation:

  • Nextflow: In active use with gRED FDFM (Foundational Data for Foundational Models), and omics workflows

  • Argo Workflows: Successfully used in the Spring Sciences platform, and is the default executor in the pRED-MLOps’ Kubeflow Pipelines. Note: the Dagster vendor (inherently biased) said they have seen customers migrate from Argo to Dagster, since Argo is infra-centric and best “for DevOps engineers but becomes a bottleneck for data engineers, analytics engineers, or ML practitioners who want Pythonic, declarative workflows and local testing”.

🌈 Consequences

Pros:

  • Provide immediate value to existing Dagster users (e.g. AI4DD) through Dagster+ specific (e.g. branch deployments, launcher role) functionality they have requested

  • Demonstrate whether Dagster+ RBAC functionality will enable scaling Dagster to a wider set of users, and for use cases where auditing is required

  • Build evidence to support a CoE-wide decision on the value of Dagster, and a decision on whether to have a unified orchestration platform versus when to have use-case, data domain specific platforms

  • Validate Dagster could be integrated with data catalog, lineage, and quality tools to provide a more robust, consistent framework for data pipelines to support our CoE Data Ecosystem

Cons:

  • Higher cost vs. open source alternatives (e.g. Dagster, Airflow, Kubeflow, Nextflow, Argo)

  • Vendor dependence

  • Perception risk that this signifies Dagster has already been selected as a CoE standard

  • The learning curve with Dagster’s asset-based approach, though existing users have reported it’s relatively easy to learn

Neglected Options based on desired outcomes:

  • Delay purchase of Dagster+: We could have postponed procuring Dagster+ until selecting a CoE standard, but this provides valuable learning at low commitment and provides value for AI4DD

  • Standardize immediately: Dagster+ purchase is low commitment through existing 2025 budget, and enables us to gather further evidence in our new CoE org

Implementation considerations

Why Dagster+ over Dagster OSS?

Container Orchestration Platform Deployment

  • Will run on Lightship, the replacement for Nostromo, as part of the gRED Cloud Foundation EKS platform. Dagster OSS already runs on Nostromo, which will be decommissioned by end of 2025

  • Technical Support & Operations team (in DDC-Solutions) will own infrastructure operations, including user management

Evaluation Use Cases

To demonstrate impact and guide potential CoE-wide decision: * Data pipeline: via gRED Data Ecosystem, 1-2 new pipelines built on Dagster+, plus design (and preferably PoC) to validate integrations with catalog, lineage, and quality tools * MLOps: 1-2 ML workflows to test Dagster+ alongside existing Kubeflow

Execution will be collaborative across legacy gCS and D&A orgs to ensure input into a CoE-wide recommendation.

🤝 Outcome / Decision

A formal checkpoint will occur in Q1 2026 led by Enterprise Architecture to review progress and evaluation against agreed upon criteria, rather than waiting until end of term:

  • Eval Criteria

    • Ease of Use

    • Adoption

    • Validation against data pipeline use cases, including demonstrating (by design or implementation) integration with catalog and data lineage

    • Validation against MLOps use cases

    • Financial viability (with forecast) for broader organization adoption

    • Migration effort required for existing solutions and platforms to potentially consolidate towards Dagster+

    • Security compliance status (for being capable to rollout to boarder organization)

    • Operational efficiency

  • Outcome of checkpoint

    • Clarify which use cases Dagster+ is best suited for, and which are better served by other platforms

    • Decide whether Dagster+ should be established as a CoE orchestration standard (primary or one of several)

📚 Resources