Real World Big Data in Azure
The Big Data components of Azure let you build solutions which can process billions of events, using technologies you already know. In this course, we build a real world Big Data solution in two phases, starting with just .NET technologies and then adding Hadoop tools.
What you'll learn
How do you make sense of Big Data? When you’re receiving 100 million events per hour and you need to save them all permanently, but also process key metrics to show real-time dashboards, what technologies and platforms can you use? This course answers those questions using Microsoft Azure, .NET, and Hadoop technologies: Event Hubs, Cloud Services, Web Apps, Blob Storage, SQL Azure, and HDInsight. We build a real solution that can process ten billion events every month, store them for permanent access, and distill key streams of data into powerful real-time visualizations.
Table of contents
- Introducing Big Data in Azure 2m
- Is This Course for Me? 2m
- Building the Phase 1 Solution 3m
- Evolving the Phase 2 Solution 3m
- Introducing the Telemetry API 2m
- Demo: EventsController 4m
- Key Features of the Telemetry API 3m
- Zipping the Event Batch 2m
- Demo: GZipToJsonHandler 5m
- Testing the API 2m
- Demo: IEventSender 4m
- Module Summary 3m
- Module Introduction 2m
- Introducing Event Hubs 2m
- Configuring Event Hubs 3m
- Creating Event Hubs 3m
- Automating Event Hub Creation 3m
- Sending Events 3m
- Demo: EventHubEventSender 4m
- Batching Events 3m
- Demo: EventBatchIterator 4m
- Batching and Testing 2m
- Demo: Acceptance Tests 3m
- Understanding Throughput Units 4m
- Demo: Deploy, Monitor & Scale 4m
- Next Steps for Deployment & Monitoring 2m
- Isolating the Event Hub 2m
- Module Summary 3m
- Module Introduction 2m
- Aim of Deep Storage 3m
- Concurrency and Partitions 2m
- Demo: IEventProcessor 4m
- Competing Consumers 3m
- Buffering Output 3m
- Demo: DeepStorageEventProcessor 3m
- Level 1: In-Process Memory 3m
- Demo: MemoryEventStore 3m
- Level 2: Local Disk 3m
- Demo: DiskEventStore 4m
- Level 3: Blob Storage 2m
- Demo: BlobStorageEventStore (Part 1) 3m
- Demo: BlobStorageEventStore (Part 2) 3m
- Timed Flushing 3m
- Demo: Deep Storage Worker Role 3m
- Scaling Read with Throughput Units 2m
- Module Summary 2m
- Module Introduction 2m
- About Pig 3m
- Demo: AzCopy and Pig 4m
- Importing Libraries 3m
- Demo: Pig and JSON Data 3m
- HDInsight and Pig 3m
- Demo: Running Pig Scripts in HDInsight 5m
- Streaming Data to Commands 3m
- Demo: Streaming from Pig to .NET 5m
- Visualizing Output from Pig 3m
- Demo: Visualization with D3.js 3m
- Module Summary 3m
- Module Introduction 2m
- Modeling Event Metrics 4m
- Demo: Building the Data Model 4m
- Refactoring the Event Receiver 4m
- Demo: Core.EventProcessor 4m
- Dependency Injection and the EventProcessorHost 3m
- Buffering in the Event Handler 2m
- Demo: ModelBuffer of T 4m
- Handling Event Metrics 3m
- Demo: EventMetricsHandler 5m
- Running On-Premise 2m
- Demo: RealTime.WorkerRole 4m
- Running in Azure 2m
- Demo: Azure Deployment 4m
- Module Summary 4m
- Module Introduction 2m
- Demo: Installing Dashing.net 3m
- About Widgets 3m
- Demo: The Number Widget 3m
- About Scheduled Jobs 3m
- Demo: Event Count Job 4m
- How Dashing Pushes Updates 3m
- Adding New Widgets 2m
- Demo: The Pie Widget 3m
- Running Dashing on Azure 2m
- Demo: Deploying Azure Web App 3m
- Demo: Securing the Web App 3m
- Module Summary 4m
About the author
Elton is a 10-time Microsoft MVP, author, trainer and speaker. He spent most of his career as a consultant working in Microsoft technologies, architecting and delivering complex solutions for industry leaders. He has delivered APIs on Azure serving millions of clients daily, Big Data solutions processing billions of events weekly, and cutting-edge solutions powered by containers. Elton's experience with .NET goes from .NET 1.0 running on Windows Server, right up to .NET Core running on Linux. Wh... more