[Free] Data Engineering Bootcamp: Python, Dbt, Airflow &Amp; Sql

Build end-to-end data pipelines with APIs, S3, dbt, Airflow, data quality, CI/CD and hands-on production scenarios.

Added on November 13, 2026 Development 5 min read

What you’ll learn

• Ingest a production database incrementally and idempotently, with watermarks and upserts
• Ingest a REST API properly: tokens, pagination, rate limits (429) and retries
• Ingest event files from S3-style object storage exactly once, and survive schema drift
• Protect pipelines with data contracts that fail loudly when a source changes
• Model data with dbt: staging, star schemas, slowly changing dimensions (SCD2) and snapshots
• Test data quality with dbt tests and source freshness, and stop bad data before dashboards
• Orchestrate everything with Apache Airflow: schedules, retries, alerts and recovery
• Make models fast with incremental builds and indexes, and ship changes with CI/CD
• Handle a real production incident, and present the project to recruiters

Requirements

• A computer that can run Docker Desktop, with about 4 GB of free memory
• Basic SQL, and basic comfort with a terminal. Python helps but isn't required

Description

Want to become a Data Engineer? Start building real data pipelines.

This practical, project-based course introduces modern data engineering through a complete learning journey covering data ingestion, transformation, modelling, orchestration, testing and production-style operations. Rather than learning disconnected tools, you will explore how different technologies work together to build reliable data pipelines.

At the centre of the course is the FreshCart Data Platform, a practical data engineering project that demonstrates how data moves from multiple sources through processing and transformation to analytical reporting.

What You’ll Learn

You will develop practical skills in:

• Understanding the architecture of an end-to-end data engineering platform

• Setting up a hands-on data engineering environment using Docker Desktop

• Implementing incremental database ingestion using watermarks and upserts

• Understanding data contracts and their importance in reliable pipelines

• Ingesting data from REST APIs using authentication tokens and pagination

• Handling API rate limits, retries and ingestion failures

• Working with S3 event files and understanding schema drift

• Creating and organising dbt sources, models and dependencies

• Understanding data lineage and how transformations connect

• Designing star schemas for analytical reporting

• Exploring Slowly Changing Dimensions Type 2

• Applying dbt tests and source freshness checks

• Orchestrating data pipelines using Apache Airflow

• Understanding pipeline retries, alerts and recovery

• Working with incremental models and performance optimisation

• Exploring CI/CD using linting, unit tests and GitHub Actions

• Investigating a simulated production schema-change incident

• Completing a final dashboard and portfolio capstone project

Build the FreshCart Data Platform

Instead of completing unrelated exercises, you will explore the development of the FreshCart Data Platform. This connected project provides a realistic context for understanding how data engineers work with source systems, data ingestion, transformations, analytical models and production workflows.

You will follow the data engineering lifecycle from initial setup through to the final dashboard and portfolio deliverable.

Part 1: Ingest Data from Multiple Sources

Every data pipeline begins with data ingestion. You will explore three important ingestion patterns.

Incremental database ingestion: Learn how data engineers avoid repeatedly processing entire datasets. You will explore watermarks, upserts and data contracts.

REST API ingestion: Discover how applications and services expose data through APIs. You will work with authentication tokens, pagination, rate limits and retry strategies.

Data lake ingestion: Explore ingestion from S3 event files and investigate schema drift, a common challenge that occurs when incoming data structures change unexpectedly.

Part 2: Transform Data with dbt and SQL

Collecting data is only the beginning. Before data can support reliable analytics, it often needs to be transformed, organised and validated.

You will explore dbt and learn how it supports structured data transformation workflows. Topics include dbt fundamentals, sources, models, dependencies and data lineage. You will also explore dimensional modelling, star schemas, Slowly Changing Dimensions Type 2, dbt tests and source freshness checks.

These skills provide a foundation for developing trustworthy analytical datasets.

Part 3: Orchestrate Pipelines with Apache Airflow

A data pipeline becomes more useful when its tasks can run in a coordinated and repeatable way. You will explore how Apache Airflow supports workflow orchestration, including scheduling tasks, managing retries, handling failures and recovering from interrupted workflows.

You will also examine incremental models and performance considerations that help make data processing more efficient.

Part 4: Explore CI/CD for Data Engineering

Modern data engineering teams need ways to validate changes before introducing them into important workflows. You will explore how Continuous Integration and Continuous Delivery practices apply to data pipelines.

The course introduces code linting, unit testing, automated validation and GitHub Actions. You will see why testing and automation are valuable for improving the reliability and maintainability of data engineering projects.

Part 5: Investigate a Production-Style Pipeline Failure

What happens when a data pipeline breaks at 2 a.m.? This is one of the most valuable practical scenarios in the course.

In the lesson “Incident: The 2 a.m. Schema Change,” you will investigate a simulated production problem involving an unexpected schema change. You will apply your knowledge of pipeline operations, data quality and recovery to understand how such incidents affect data workflows.

Building a pipeline is only part of the job. Understanding how to investigate and recover from failures is equally important.

Final Capstone: Dashboard and Portfolio

At the end of the course, you will bring together the concepts explored throughout the FreshCart project. The final capstone focuses on the dashboard and your data engineering portfolio.

You will finish with a practical project that demonstrates your understanding of the end-to-end data pipeline and can support your continued learning and portfolio development.

Who Is This Course For?

This course is ideal for aspiring Data Engineers, Python and SQL developers, Data Analysts, aspiring Analytics Engineers, software developers, IT professionals, career changers, computer science and IT students, and project-based learners.

You do not need previous professional data engineering experience to begin. However, basic familiarity with Python, SQL and general programming concepts will make the practical lessons easier to follow.

You will need a computer capable of running Docker Desktop, an internet connection, basic computer and file-management skills, and a willingness to practise and troubleshoot technical problems.

Why Choose This Bootcamp?

• Complete an end-to-end data engineering project

• Work with Python, SQL, dbt, Apache Airflow, Docker, REST APIs, S3 concepts and GitHub Actions

• Learn practical ingestion patterns, including incremental ingestion, API pagination, retries and schema drift

• Develop data modelling and quality skills using star schemas, SCD2, dbt tests and source freshness checks

• Understand production-style engineering, orchestration, performance, CI/CD and pipeline recovery

• Investigate a realistic schema-change incident

• Build a portfolio-focused dashboard and capstone project

• Follow a structured learning journey from ingestion to production operations

Enrol today and start building your data engineering skills with Python, SQL, dbt and Apache Airflow.

Author(s): Bluelime Learning Solutions