How to Install Python Dependencies from Private Repositories in AWS MWAA

Discover how to effectively install Python dependencies from private repositories using AWS MWAA for your ETL processes without the burden of managing Airflow environments.
---
This video is based on the question https://stackoverflow.com/q/71469268/ asked by the user 'Manuel Martinez' ( https://stackoverflow.com/u/8998223/ ) and on the answer https://stackoverflow.com/a/73146294/ provided by the user 'ferrouswheel' ( https://stackoverflow.com/u/272238/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: AWS | MWAA Dependency in private repo

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Simplifying Python Dependencies in AWS MWAA: Installation from Private Repositories

If you’re working with AWS Managed Workflows for Apache Airflow (MWAA) and trying to integrate private Python dependencies from repositories like GitHub, you might face some challenges. Many developers are shifting towards AWS MWAA to streamline their ETL (Extract, Transform, Load) processes while minimizing the complexity of managing their own Airflow servers. But what if you need to install packages that are not publicly available on PyPI and are hosted in your private GitHub repositories?

This guide aims to address this common issue and guide you step-by-step on how to install Python dependencies from private repositories in AWS MWAA.

The Problem: Installing Dependencies from Private Repositories

AWS MWAA provides a managed service for running Apache Airflow workflows, but it comes with certain limitations, particularly when accessing private resources. When you want to install Python dependencies from private repositories, the challenge lies in being able to authenticate and access those repositories securely.

Common Questions Include:

How can we securely retrieve packages stored in private GitHub repositories?

Is there a way to use a standard requirements.txt file while maintaining security?

What are the best practices for handling dependencies in a privately hosted environment?

The Solution: Using .whl Files in plugins.zip

To install Python packages from private repositories in AWS MWAA, the recommended approach is as follows:

Step 1: Build the Package as a Wheel (.whl)

Create a Wheel Distribution: Instead of directly referring to your private repo in the requirements.txt, compile your package into a .whl file. This can be done using the command:

[[See Video to Reveal this Text or Code Snippet]]

This command creates a distributable package format that is easy to install.

Step 2: Upload the .whl File to MWAA

Place the .whl File in the plugins.zip:

AWS MWAA allows you to upload a plugins.zip file that contains custom plugins and dependencies.

Include your built .whl file inside this plugins.zip to make it accessible to your MWAA environment.

Step 3: Restart Your MWAA Environment

Deploy and Test: After you have uploaded the updated plugins.zip file, restart your MWAA environment. This will ensure that your new dependencies are loaded properly.

Benefits of This Approach

Simplified Management: By using .whl files, you eliminate potential issues related to authentication tokens and private repositories.

Portability: Wheel files are self-contained, making them portable and easy to manage within the AWS ecosystem.

Version Control: You can manage your dependencies and their versions effectively through the wheel distribution process.

Alternative Options: Using S3

While the above solution is ideal, some might consider alternative methods such as:

Copying Repositories to S3: As you mentioned, one workaround is to host your code or packages in an S3 bucket. However, this approach can be cumbersome and does not leverage the benefits of pre-built distributions.

Conclusion: Streamlining Your Dependency Management

Managing dependencies in AWS MWAA does not have to be complicated. By following the suggested method of building .whl files and placing them in a plugins.zip, you can efficiently manage your private Python dependencies without the hassle of maintaining a separate authentication mechanism for private repositories. This not only enhances security but also aligns well with the Fast, Scalable, and Efficient features that AWS offers.

Thank you for reading! If you have any further questions or need additional insights on AWS MWAA, feel free to reach out