![]() ![]() The first update I make is to make Airflow pick up changes more quickly by updating the dag_dir_list_interval from the default of 5 minutes (300) to 5 seconds. I make a couple of changes to my airflow.cfg to be more responsive during the development/testing cycle, as well as also enabling the Airflow REST API. Make sure you use credentials that have limited access to only those services you need on development environments. ![]() REMEMBER Never ever share this file or put it anywhere that anyone can see. If you are going to be interacting with AWS services, then you will need to update the ".env.localrunner" file with your credentials. Note! As you use mwaa-local-runner, every time you make changes to the Airflow configuration, you might need to re-build the image. Before we build this image however, I want to make a few changes. The first time you use mwaa-local-runner you need to build the image that will then be used every time you start mwaa-local-runner. You can also tailor this to suit your developer flow, and I made a few tweaks for my particular setup which I will share here.Įnter fullscreen mode Exit fullscreen mode It makes it super easy to get Apache Airflow up and running. Mwaa-local-runner is a project that provides a configuration parity with the AWS MWAA managed service. Docker and Docker-compose running on your local machine.VSCode - I am running version 1.75.1 on my Mac.You will need the following to get this up and running Once that is up and running, I will show you the VSCode plugins and how they work. I will walk you through setting up mwaa-local-runner, including making a few changes that will enable the VSCode plugins to work. In this post I will show you how you can configure mwaa-local-runner, an open source project that provides you with an easy way to get a local Apache Airflow environment up and running (that is configuration wide, aligned to the Amazon Managed Workflows for Apache Airflow service MWAA), together with some VSCode plugins. This post was inspired by a tweet from Kaxil Naik who was asking about what features developers are looking for when using VSCode and Pycharm and Apache Airflow. In this short post, I wanted to highlight how you can use VSCode plugins to work with a local running instance of Apache Airflow to improve the developer experience. ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |