We built an ETL pipeline and data warehouse for EdCast using AWS Glue and S3. This article lists key learnings. The three ways to build and test Glue ETL scripts are the AWS-hosted development endpoint, a local notebook connected to that endpoint, and local development with the AWS Glue ETL library. The library is the reduced-cost option and supports unit testing.
AWS Glue Development Environment
Develop and test Glue ETL scripts in three ways:
- AWS-hosted development endpoint and notebook
- Local environment with development endpoint and Zeppelin notebook server
- Local development using the ETL library
Remote development endpoint and notebooks
The AWS development endpoint is a paid, managed Glue environment for building and testing ETL scripts. It includes Apache Spark, Glue libraries, and network configuration for secure access from Jupyter notebooks.
AWS Glue Development Endpoints provision EC2 instances with Jupyter Notebook. Users use these instances to write and test ETL code for Glue jobs.
See the AWS documentation for development endpoint details.
https://docs.aws.amazon.com/glue/latest/dg/console-development-endpoint.html
Advantages
- Simple setup and operation
- The development endpoint mirrors the AWS Glue production environment. This setup simplifies development and testing.
Limitations
- High cost. The development endpoint costs $1,500 per month (as of March 22, 2020). Additional costs apply for the EC2 instance that hosts the notebook server.
- Development requires an internet connection to the notebook server.
- The platform lacks a streamlined method for writing unit tests.
Use the development endpoint and Jupyter Notebook in the local environment
Run the Jupyter Notebook server in the local environment. Connect to the development endpoint via an SSH tunnel. Update the development endpoint with your specific RSA SSH public key. Select Rotate SSH Keys on the Development Endpoints page.
The Docker version of Jupyter 0.8.2 does not work with the Glue development endpoint due to bugs. Install the binary locally.
Advantages
- No separate notebook server is required
- Runs on local machines. Network latency causes UI instability.
Limitations
- High cost. The development endpoint costs $1,500 per month.
- Slow development cycle. The endpoint runs in a remote environment.
- No simple method for writing unit tests.
Local development with the AWS Glue ETL library
AWS released the AWS Glue library for local development. This library integrates Glue ETL jobs with the Maven build system for building and testing.
Developers can build ETL logic in Zeppelin, PyCharm Professional 2019.3, or Visual Studio Code. We use PySpark for ETL scripts. We develop these scripts in PyCharm. PyCharm supports local execution and debugging. It also supports remote debugging.
Benefits
- The tool set simplifies configuration and reduces the time required for development and testing.
- Reduced Cost
- Supports Unit Testing
For instructions on local setup and execution of Glue jobs, see:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html
The next article details the steps to configure PyCharm with the Glue ETL library for local debugging.
For help building ETL pipelines on the cloud, see our data engineering services.
Related reading: how to set up a data pipeline on GCP and change data capture for Snowflake, PostgreSQL, MySQL and MongoDB.