[100% Off] Aws Data Engineering Project: End-To-End Rds To S3 With Glue
Build a real-world incremental pipeline with AWS Glue, PySpark, Parquet, Glue Data Catalog, Workflows and Triggers.
What you’ll learn
- Build an end-to-end incremental pipeline from Amazon RDS PostgreSQL to Amazon S3 using AWS Glue.,Implement watermark-based ingestion to load only new and updated records from multiple source tables.,Store data as Parquet
- catalog it with AWS Glue Crawler
- and query it using Amazon Athena.,Schedule and orchestrate the pipeline using AWS Glue Workflows and Triggers.
Requirements
- Basic familiarity with SQL and Python will help you get the most out of this course,A computer with internet access and an AWS account.,No prior AWS or AWS data engineering experience is required.
Description
Build a realistic AWS data engineering project that ingests healthcare data from Amazon RDS PostgreSQL into an Amazon S3 data lake using AWS Glue.
You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, making design decisions, setting up AWS, implementing the solution, validating the output, and scheduling repeatable runs.
In this project, you will:
Prepare the source: Set up a PostgreSQL database in Amazon RDS
Build the Glue job: Ingest multiple source tables using AWS Glue and PySpark
Load incrementally: Use watermarks to process new and updated records
Store the data: Write the output to Amazon S3 as Parquet files
Track each run: Add ingestion metadata for auditing and validation
Catalog the data: Use AWS Glue Crawlers and the Glue Data Catalog
Validate the output: Query the ingested data using Amazon Athena
Schedule the pipeline: Use Glue Workflows and Triggers for repeatable runs
More than isolated service demonstrations
You will build the project inside your own AWS account using the provided source code, healthcare sample data, configuration files, and infrastructure setup scripts.
The focus is not only on getting the pipeline to run. You will understand why each AWS service is used, how the components work together, and how the pipeline handles new and updated source records during subsequent runs.
By the end of the course, you will have a complete AWS data engineering project that you can practice, adapt for your portfolio, and explain clearly during interviews.
This course contains a promotion.
