Optimize Workflows with Apache Airflow Operators
This course will teach you to create efficient DAGs using branching, file sensors, PythonOperators, and DummyOperators, and to create custom operators for logging, enabling you to handle complex data processing with ease.
What you'll learn
Apache Airflow offers a wide variety of operators to help you optimize your workflows.
In this course, Optimize Workflows with Apache Airflow Operators, you’ll gain the ability to streamline your data pipelines and enhance workflow efficiency using various Airflow operators.
First, you’ll explore how to create a DAG that performs conditional data processing using a branching operator and checks for file availability with a FileSensor.
Next, you’ll discover how to perform data validation with a PythonOperator, group tasks using a DummyOperator, and utilize Airflow variables for dynamic processing.
Finally, you’ll learn how to generate both aggregated and filtered reports, write them out to an output folder, and then create your own custom operator to clean data that you will use in your workflow.
When you’re finished with this course, you’ll have the experience working with a variety of Airflow operators needed to optimize and maintain complex data workflows efficiently.
Table of contents
About the author
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing ... more