DolphinScheduler
Apache DolphinScheduler is the modern data workflow orchestration platform with powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available `out of the box`
Stars
11.4K
Forks
4.25K
Open issues
1.04K
Closed issues
5.99K
Last release
7 months ago
Last commit
5 months ago
Watchers
11.4K
Total releases
49
Total commits
8.13K
Open PRs
99
Closed PRs
6.14K
- Project Website
- https://dolphinscheduler.apache.org
- Platform
- License
- apache-2.0
- Category
- Technology
- Python Typescript javascript Shell Dockerfile Smarty SCSS PLpgSQL HCL
- Offers premium version?
- NO
- Proprietary?
- NO
- About
-
Apache Dolphinscheduler
AboutApache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code. It is also provided powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available out of the box
The key features for DolphinScheduler are as follows:
- Easy to deploy, provide four ways to deploy which including Standalone, Cluster, Docker and Kubernetes.
- Easy to use, workflow can be created and managed by four ways, which including Web UI, Python SDK, Yaml file and Open API
- Highly reliable and high availability, decentralized architecture with multi-master and multi-worker, native supports horizontal scaling.
- High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day
- Cloud Native, DolphinScheduler supports orchestrating multi-cloud/data center workflow, and supports custom task type
- Versioning both workflow and workflow instance(including tasks)
- Various state control of workflow and task, support pause/stop/recover them in any time
- Multi-tenancy support
- Others like backfill support(Web UI native), permission control including project, resource and data source
- For quick experience
- Want to start with standalone
- Want to start with Docker
- For Kubernetes
- For Terraform
-
Homepage: Project and workflow overview, including the latest workflow instance and task instance status statistics.
-
Workflow Definition: Create and manage workflow by drag and drop, easy to build and maintain complex workflow, support bulk of tasks out of box.
-
Workflow Tree View: Abstract tree structure could clearer understanding of the relationship between tasks
-
Data source: Manage support multiple external data sources, provide unified data access capabilities for such as MySQL, PostgreSQL, Hive, Trino, etc.
-
Monitor: View the status of the master, worker and database in real time, including server resource usage and load, do quick health check without logging in to the server.
Follow this guide to report your suggestions or bugs.
ContributingThe community welcomes everyone to contribute, please refer to this page to find out more: How to contribute, find the good first issue in here if you are new to DolphinScheduler.
CommunityWelcome to join the Apache DolphinScheduler community by:
- Join the DolphinScheduler Slack to keep in touch with the community
- Follow the DolphinScheduler Twitter and get the latest news
- Subscribe DolphinScheduler mail list, users@dolphinscheduler.apache.org for user and dev@dolphinscheduler.apache.org for developer
DolphinScheduler enriches the CNCF CLOUD NATIVE Landscape.
Alternative Projects
-
Amazon Elastic Container Service (Amazon ECS) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run and scale containerized applications on AWS.
-
A multi-cluster batch queuing system for high-throughput workloads on Kubernetes.374
-
Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale.3K
-
Multi-tenancy and policy-based framework for Kubernetes.1.33K
-
[CNCF Sandbox Project] Managing your Kubernetes clusters (including public, private, edge, etc.) as easily as visiting the Internet ⎈1.26K
-
Clusterpedia is used for complex resources search across multiple clusters, support simultaneous search of a single kind of resource or multiple kinds of resources existing in multiple clusters.682
-
Cloud Native Control Planes7.86K
-
A toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.3.14K
-
Fluid is an orchestration platform for elastic data abstraction and acceleration in cloud native environment.1.46K
-
iSoftStone Multi-cloud Management System is an enterprise-level cloud management platform applied in multiple public clouds and public-private hybrid cloud scenarios. It provides consistent self-service and operation management capabilities on heterogeneous resources which include centralized resource management and control, automated resource delivery, intelligent operation analysis and O&M management to reduce the costs of hybrid cloud management.
-
Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration3.67K
-
QoS based scheduling system for hybrid orchestration workloads on Kubernetes, bringing workloads the best layout and status.1.05K
-
kube-green is a simple k8s addon that automatically shuts down (some of) your resources when you don't need them.772
-
kube-rs is the core Rust ecosystem for building applications against Kubernetes2.44K
-
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications104K
-
Kured (KUbernetes REboot Daemon) is a Kubernetes daemonset that performs safe automatic node reboots when the need to do so is indicated by the package management system of the underlying OS1.98K
-
Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.14.1K
-
Contains useful documentation on the OCM project. Report here if you found any issues in OCM.638
-
The open source Cloud & Edge Computing Platform bringing real freedom to your Enterprise Cloud 🚀1.07K
-
The easiest way to orchestrate and observe your data pipelines13.5K
-
StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html5.77K
-
A Cloud Native Batch System (Project under CNCF)3.47K
-
Project homepage. wasmCloud allows for simple, secure, distributed application development using WebAssembly actors and capability providers.1.09K
-
Apache ZooKeeper11.7K
-
AWS Cloud Map is a cloud resource discovery service. With Cloud Map, you can define custom names for your application resources, and it maintains the updated location of these dynamically changing resources. This increases your application availability because your web service always discovers the most up-to-date locations of its resources.
-
CoreDNS is a DNS server that chains plugins11.3K
-
Distributed reliable key-value store for the most critical data of a distributed system45.1K
-
A cloud native Kubernetes Global Balancer680
-
A High Performance Metadata System for Kubernetes693
-
an easy-to-use dynamic service discovery, configuration and service management platform for building cloud native applications.28K
-
AWS Service registry for resilient mid-tier load balancing and failover.12.1K
-
Apache Avro is a data serialization system.2.64K
-
CloudWeGo is ByteDance's open source Golang-centric middleware that can be used to quickly build enterprise-class cloud native architectures.6.4K
-
Apache Dubbo is a high-performance, java based, open source RPC framework.39.6K
-
go-zero is a web and rpc framework written in Go. It's born to ensure the stability of the busy sites with resilient design. Builtin goctl greatly improves the development productivity.26.5K
-
The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)39.6K
-
Kratos is a microservice-oriented governance framework implements by golang, which offers convenient capabilities to help you quickly build a bulletproof application from scratch.21.8K
-
SOFARPC is a high-performance, high-extensibility, production-level Java RPC framework.3.76K
-
High performance, low latency, lightweight enterprise-level RPC system, which supports Baidu bRPC, Tencent tRPC, thrift protocols.1.77K
-
TARS is a high-performance RPC framework that provides an integrated solution of microservice governance.9.71K
-
Avi Networks drives automation and intelligence across multi-cloud environments with intent-based application services, including LB and WAF
-
Open-source layer 7 load balancer derived from proprietary Baidu FrontEnd5.99K
-
Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go50.9K
-
Citrix ADC is an application delivery and load balancing solution that provides a high-quality user experience for your web, traditional, and cloud-native applications regardless of where they are hosted.
-
Contour is a Kubernetes ingress controller using Envoy proxy.3.54K
-
Cloud-native high-performance edge/middle/service proxy23.1K
-
F5 Networks provides application security and delivery tools.
-
Gimbal is an ingress load balancing platform capable of routing traffic to multiple Kubernetes and OpenStack clusters. Built by Heptio in partnership with Actapio.664
-
HAProxy Load Balancer's development branch (mirror of git.haproxy.org)4.15K
-
A network load-balancer implementation for Kubernetes using standard routing protocols6.29K
-
The Cloud-Native Network Proxy Platform4.29K
-
Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more.13K
-
An official read-only mirror of http://hg.nginx.org/nginx/ which is updated hourly. Pull requests on GitHub cannot be accepted and will be automatically closed. The proper way to submit changes to nginx is via the nginx development mailing list, see http://nginx.org/en/docs/contributing_changes.html19.5K
-
Load Balancer Implementation for Kubernetes in Bare-Metal, Edge, and Virtualization1.46K
-
High Performance Web Platform Based on Nginx and LuaJIT11.8K
-
Pipy is a programmable proxy for the cloud, edge and IoT. It's written in C++, which makes it extremely lightweight and fast. It's also fully programmable by using PipyJS, a tailored version from the standard JavaScript language.654
-
Sangfor CNAD is a cloud native application proxy, that provides layer7&layer4 load balance service, high performance TLS encryption and decryption resource pool, and Ingress&Loadbalancer for kubernetes.
-
A powerful flow control component enabling reliability, resilience and monitoring for microservices. (面向云原生微服务的高可用流控防护组件)21.6K
-
An HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress2.96K
-
Snapt provides future-proof load balancing and security solutions that are cloud-neutral and platform-agnostic.
-
A Kubernetes media gateway for WebRTC. Contact: info@l7mp.io579
-
A distribution of Nginx with some advanced features12.4K
-
Akana is a full lifecycle API management platform that supports multicloud deployments with industry-leading security.
-
Azure API Management is a hybrid, multi-cloud API, full lifecycle management platform for APIs across all environments. It allows customers to self-host API gateways as containers on Kubernetes.113
-
A Cloud Native traffic orchestration system5.58K
-
open source Kubernetes-native API gateway for microservices built on the Envoy Proxy4.19K
-
EnRoute Ingress API Gateway - Connectivity and Security for your Microservice at Kubernetes Ingress using one helm command. Built on Envoy Proxy.188
-
The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy3.91K
-
Gravitee.io - OpenSource API Management94
-
Hango API Gateway, build on Envoy & Istio.442
-
Next-generation Cloud Native Gateway | 下一代云原生网关1.93K
-
Ultra performant API Gateway with middlewares. A project hosted at The Linux Foundation5.83K
-
Specific Layer7 Gateway for kube-apiserver414
-
Kusk Gateway is a self-service API gateway powered by OpenAPI and Envoy. Kusk Gateway is built and maintained by Kubeshop.242
-
Create low-latency, interactive user experiences for stateless microservices.571
-
Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols8.96K
-
Choreo Connect is a cloud-native, open-source, and developer-centric API gateway.276
-
Aeraki Mesh allows you to manage any layer-7 traffic in a service mesh703
-
AWS App Mesh is a service mesh that provides application-level networking to make it easy for your services to communicate with each other across multiple types of compute infrastructure. App Mesh standardizes how your services communicate, giving you end-to-end visibility and ensuring high-availability for your applications.
-
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.27.4K
-
A service mesh implementation for connecting, control, and observe services in spring-cloud.496
-
Glasnostic makes modern cloud applications resilient by shaping how systems interact, automatically and in real-time.
-
The Service Mesh Orchestration Platform1.01K
-
greymatter.io uses the service mesh ecosystem to create connected intelligence and intervention for next-gen modern infrastructure.
-
Simplify observability, traffic management, security, and policy with the Istio.34K
-
🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.3.35K
-
Ultra light, ultra simple, ultra powerful. Linkerd adds security, observability, and reliability to Kubernetes, without the complexity.10K
-
Use eBPF to speed up your Service Mesh like crossing an Einstein-Rosen Bridge.696
-
Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.2.59K
-
Universal cloud-native microservice governance specification (微服务治理标准)688
-
Sermant a proxyless service mesh solution based on Javaagent.807
-
Service Mesh Interface1.05K
-
Standardizing Service Mesh Value Measurement263
-
An intelligent ServiceMesh manager based on Istio406
-
Application connectivity platform.
-
Traefik Mesh - Simpler Service Mesh1.94K
Subscribe to Open Source Businees Newsletter
Twice a month we will interview people behind open source businesses. We will talk about how they are building a business on top of open source projects.
We'll never share your email with anyone else.