Virtualization, Docker, and Kubernetes for Data Engineering

Duke University via Coursera

Coursera based on 38 ratings

Likes

Overview

Virtualization, Docker, and Kubernetes for Data Engineering" is a comprehensive course designed to equip data engineers with essential skills in containerization and orchestration technologies. This course focuses on how virtualization, Docker, and Kubernetes streamline data workflows, improve scalability, and enable efficient deployment of data applications. Learners start by understanding the fundamentals of virtualization and how it creates isolated environments for running applications. The course then dives into Docker, teaching how to build, manage, and deploy lightweight, portable containers that package data...

Syllabus

Virtualization Theory and Concepts

In this module, you will learn about the fundamentals of virtualization, exploring its various aspects such as hardware utilization and scaling applications. You will start by understanding what virtualization is and delve into the concept of virtual machines. Through the introduction of Virtual Box and a hands-on demo, you will gain a practical understanding of how virtual machines work and their benefits. Additionally, you will explore container concepts, focusing on Docker as a key containerization tool. Through an introduction to Docker and its architecture, you will learn how to scale applications using containers, providing a comprehensive overview of virtualization and its practical applications. To apply your newfound knowledge, you will be assessed through a series of hands-on exercises involving the creation and management of virtual machines and containers, demonstrating your ability to effectively utilize virtualization technologies.

Using Docker

In this module, you will learn how to effectively work with the Docker client, create volumes, and run databases in containers, gaining hands-on experience in managing containerized applications. You will also explore how to use the Docker command line for tasks such as building images and working with Dockerfiles, enabling you to package your software efficiently. You'll get a chance to study real-life Dockerfile examples and consult the Dockerfile reference for best practices. Furthermore, you will dive into orchestration with Docker Compose, learning how to manage multi-container applications using Compose. As an extension to this, you will be introduced to Airflow, a workflow management platform, and learn how to integrate it with Docker Compose for a seamless automation experience.

Kubernetes: Container Orchestration in Action

In this module, you will embark on a comprehensive journey into Kubernetes, the cornerstone of modern container orchestration. You'll begin by grasping Kubernetes key concepts, cluster architecture, and service deployments. The advantages of cloud development environments, exemplified by GitHub Codespaces, will become more clear as you explore GitHub's ecosystem and harness AI-driven coding with GitHub Copilot and OpenAI Codewhisper. The module culminates in hands-on experience as you deploy Kubernetes using Minikube within GitHub Codespaces. Gain a solid foundation in Kubernetes essentials and the power of cloud-based development, setting the stage for successful containerized application management and collaborative coding in the modern era.

Building Kubernetes Solutions

This module immerses you in the hands-on world of Kubernetes solutions. You'll start by mastering containerization, constructing FastAPI microservices, and deploying containerized applications using Azure Container Registry and Amazon Elastic Container Registry. Next, explore options for cloud-based container orchestration, featuring Google Cloud Run and AWS Copilot, and expand your coding horizons in AWS Cloud9. Finally, address critical production issues as you delve into load testing, monitoring systems, the SRE mindset for MLOps, and the art of operationalizing microservices. This module offers a comprehensive toolkit to navigate Kubernetes in real-world scenarios, combining theory and practice to prepare you for Kubernetes success.