LightningFlow is a completely integrated Airflow platform to deploy and run jobs using Apache Airflow on AWS. Within hours, you will be ready to deploy and test your DAGs, run and monitor them on the Airflow UI.
LightningFlow comes pre-integrated with all the required libraries, webserver, scheduler and worker configurations, local Spark cluster, and a postgres database. When you start the EC2 instance, all the required services start up with standard configurations allowing you to start running and testing the DAGs immediately. The DAGs run on a local Spark cluster, thus eliminating the need to create an EMR cluster.
Why use LightningFlow:
LightningFlow, combined with the vast capabilities of Apache Airflow is a solution that helps you to focus on the core objectives of centralizing your pipelines into a single environment. It is by far, the quickest and the most reliable solution to get Airflow up and running in a Production-like environment. Whether you are looking to orchestrate your Spark ETL pipelines, or migrate existing Spark/BigQuery pipelines to Apache Airflow, LightningFlow helps you to get there the very same day, without worrying about the underlying infrastructure.
LightningFlow is available on AWS and Microsoft Azure Marketplaces. Launch the instance or VM from the respective marketplaces and quickly begin exploring the fully integrated Airflow solution to run pipeline jobs/DAGs on any environment.