I am not a core-contributor to the Prefect product, but a user
Social Distance, Wear Masks, Wash Hands, and consider allyship for those that need it now more than ever. #BLM
Prefect is an alternative workflow management system designed for (more recent) modern infrastructure.
Quite similar to Airflow, but different.
# Presentation Pseudo-Code import aircraftlib as aclib from prefect import task, Flow @task def extract_reference_data(): print("fetching reference data...") return aclib.fetch_reference_data() @task def extract_live_data(): # Get the live aircraft vector data around Dulles airport dulles_airport_position = aclib.Position(lat=38.9519444444, long=-77.4480555556) ... ... ... with Flow("etl") as flow: reference_data = extract_reference_data() live_data = extract_live_data() transformed_live_data = transform(live_data, reference_data) load_reference_data(reference_data) load_live_data(transformed_live_data) # Run Workflow flow.run() # Register Workflow to Dashboard flow.register()
Task scheduling almost instant because of Prefect utilizes Dask. This can be differentiable compared to 10second wait time for another workflow management tool
Sequential Task executution without explicity management
Tasks can directly exchange data