Airflow testing is hard because of the followings

• A bare bone local airflow is not very useful because it doesn't have the connections for the services it talks to

• Deep integrations with Kubernetes further complicates things as local airflow needs to connect to Kubernetes to spin up KubernetesPodOperator

• Some testing needs to run for a longer duration and will benefit from a persistent environment

• Not all dags are designed in a way that is side-effect safe, such as delivering data to client

The solution

To show a good developer story on airflow development, we built a framework with multiple testing tiers to satisfy the needs for various testing scenarios.

• Remotely hosted environment

This is by far the most popular way for airflow development, as developers will "deploy" to a production-like airflow instance and observe their dag behavior operate in a cloud-native way. Deployment are managed by a dedicated github branch synced by git-sync sidecar container.

Such environments offers many benefits, such us persistent connection definition, collaborative testing, audit logging, and new developers and start developing from day-1 without any local setup.

To avoid the commit history from diverging, we build a synchronization service to force-push and override the entire branch off main at a pre-denfined cadence that matches the development cycle and iteration speed.

For early stage of developing, we recommend a develop environment with task scheduling turned off. Doing so will allow controlled testing and avoids excessive testing. Once the initial testing is complete, developer can decide if a more rigorous staging testing is needed in a schedule-aware environment is warranted.

• Local docker environment

While remote environment offers many benefit, it requires a git-flow within the development experience, which might not be the most convenient option. To supplement the remote environments, we also allow developers to launch local docker image of the airflow image, and allows local testing with volume mount and pulling access for the remote dev database, thus allowing users to spin up a context-rich local environment and get instant local change feedback.

• Declarative testing tier

While side effect management should always be considered in the design of an airflow pipeline, there are scenarios where exceptions are made, for example, if a client is receiving data, we just don't want the client to receive duplicate or wrong data every time someone makes a change. To address this issue, we introduced declarative dag tagging on each dag to only allow dags to run in the desirable environments, therefore greatly reduced the chance of undesirable side effect or excessive cost spending.