Monitor and Maintain Apache Airflow
This course will teach you to monitor, maintain, and troubleshoot Airflow workflows, ensuring timely execution and effective error handling with email notifications and SLA tracking.
What you'll learn
Monitoring and maintaining Apache Airflow workflows can be complex and critical for ensuring reliable data pipelines.
In this course, Monitor and Maintain Apache Airflow, you’ll gain the ability to effectively oversee and manage your Airflow workflows.
First, you’ll explore how to set up a DAG to read files and store filtered and processed data in a SQLite database, and you'll monitor the run of the workflow using the Airflow UI.
Next, you’ll discover how to configure the email provider settings in Airflow configuration to be able to send notifications for your DAG runs.
Finally, you’ll learn how to configure task-level and DAG-level SLAs, receive email notifications for SLA breaches, observe SLA misses in the Airflow UI, and configure DAG failed/retry actions.
When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to monitor, maintain, and ensure the reliability of your data workflows.
Table of contents
- Introduction and Version Check 1m
- Demo: Configuring Airflow to Send Emails 3m
- Demo: Configuring a Task-level SLA 4m
- Demo: Viewing SLA Misses in the UI and Receiving Email Notifications 3m
- Demo: Configuring DAG-level SLAs 3m
- Demo: Viewing DAG SLA Misses and Receiving Email Notifications 3m
- Demo: Sending Email Notifications for DAG Failures 4m
About the author
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing ... more