Productionalizing Data Pipelines with Apache Airflow 1
This course will teach you how to master production-grade Data Pipelines with ease within Apache Airflow.
What you'll learn
Production-grade Data Pipelines are hard to get right. Even when they are done, every update is complex due to its central piece in every organization's infrastructure. In this course, Productionalizaing Data Pipelines with Apache Airflow 1, you’ll learn to master them using Apache Airflow. First, you’ll explore what Airflow is and how it creates Data Pipelines. Next, you’ll discover how to make your pipelines more resilient and predictable. Finally, you’ll learn how to distribute tasks with Celery and Kubernetes Executors. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to make any Data Pipelines production grade.
Table of contents
- Introduction 1m
- Architecture of Apache Airflow 6m
- Demo: Installing Apache Airflow Locally 4m
- How Do We Represent a Pipeline in Airflow? 5m
- Demo: Our First DAG 5m
- Dissecting DAGs: Tasks and Operators 5m
- Demo: Creating Our Pipeline (Part 1) 4m
- Demo: Creating Our Pipeline (Part 2) 4m
- Key Takeaways and Tips 1m
- Introduction 0m
- Demo: Why Are My Tasks Sequential? 3m
- Sequential, Local and Celery Executors 5m
- Demo: Understanding Concurrency and Parallelism with Local Executor 7m
- Demo: Installing Celery Setup 5m
- Demo: Distributing Tasks with Celery Executor 2m
- A Note Regarding Airflow in Kubernetes 3m
- Key Takeaways and Tips 1m
About the author
Axel Sirota is a Microsoft Certified Trainer with a deep interest in Deep Learning and Machine Learning Operations. He has a Masters degree in Mathematics and after researching in Probability, Statistics and Machine Learning optimisation, he works as an AI and Cloud Consultant as well as being an Author and Instructor at Pluralsight, Develop Intelligence, and O'Reilly Media.