Can you feel the breeze? Data-aware scheduling and more new features in Airflow
Remember when you last used Airflow? You might have been scratching your head a few times. Recently Airflow has picked up a steady pace of releases packed with lots of interesting new features. In this talk we’ll go over them and see how they individually contribute to a new way to use Airflow.
- TaskFlow and recent improvements
- Dynamic Task Mapping
- Data-aware scheduling
- Deferrable operators
- Grid View & logging
- KubernetesPodOperator vs. KubernetesExecutor vs. KubernetesCeleryExecutor
- Official Helm chart
- Comparison with Prefect
Picture taken at the meetup