Transform Data Using PySpark
Master large-scale data manipulation and analysis with PySpark. This course covers essential techniques for handling data, creating efficient workflows, and using custom functions to streamline complex tasks.
What you'll learn
Efficient data manipulation is critical for processing large-scale datasets effectively.
In this course, Transform Data Using PySpark, you’ll gain the ability to manipulate, clean, and analyze large datasets using PySpark.
First, you’ll explore how to read and write data using various formats with schema specifications.
Next, you’ll discover how to perform advanced transformations, including grouping, joins, and window functions, as well as handle data cleaning tasks like managing missing, null, and duplicate values.
Finally, you’ll learn how to create custom functions, including UDFs, UDTFs, and vectorized UDFs, to extend PySpark's functionality for specific analytical needs.
When you’re finished with this course, you’ll have the skills and knowledge of PySpark needed to create efficient and reusable workflows for any data-driven challenge.
Table of contents
- Data Manipulations in PySpark 3m
- Demo: Reading and Writing Data with Schema Specification 4m
- Demo: Selecting and Filtering Data 3m
- Demo: Adding, Modifying, and Dropping Columns 2m
- Demo: Grouping and Aggregating Data 3m
- Demo: Performing Join Operations 4m
- Demo: Setting Operations in PySpark 3m
- Demo: Using Window Functions in PySpark 3m
About the author
Pinal Dave is an SQL Server Performance Tuning Expert and independent consultant with over 22 years of hands-on experience. He holds a Master of Science degree and numerous database certifications. Pinal has authored 14 SQL Server database books and 81 Pluralsight courses. To freely share his knowledge and help others build their expertise, Pinal has also written more than 5,800 database tech articles on his blog at https://blog.sqlauthority.com.