data-orchestration

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

Alluxio, data orchestration for analytics and machine learning in the cloud

cloud-native distributed storage

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

An open source, standard data file format for graph data storage and retrieval.

A collection of AI skills for working with Dagster

Build automated workflows, apps, and AI agents with WOML

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

Visual MLOps you host yourself: canvas or Python pipelines, training, registry, serving and drift monitoring. Export models as standalone APIs, or run them on Databricks with MLflow.

Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services

Self-hosted data platform and control plane. A Parquet lakehouse queried in-process by DuckDB, live Postgres federation in the same SQL, dbt, notebooks, and a built-in orchestrator on one machine, with no cloud tax.

Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis

Data-aware orchestration with dagster, dbt, and airbyte

Asset-first data orchestration for Elixir/BEAM. Dagster-inspired with OTP fault tolerance, LiveView dashboard, lineage tracking, checkpoint gates, and distributed execution via Oban.

This repo contains a dataset, exercises, and sample code for an end-to-end SAP BTP data-to-value bootcamp covering SAP HANA Cloud, SAP Data Warehouse Cloud, SAP Data Intelligence Cloud, and SAP Analytics Cloud.

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran

Get started with Dagster ASAP

CI/CD repository template to automate deployments of your production flows

Goldener - Make your data even more valuable - Open source data-centric orchestration for the full AI lifecycle

An operator for managing Alluxio system on Kubernetes cluster