Issue
Pipeline execution fails with a Max retry reached error.
Cause
When Sparkflows macros are used directly inside pipeline nodes, and the macro values change on every execution while the pipeline is scheduled to run at short intervals, the generated DAG code changes for each execution. This causes Airflow to continuously detect DAG changes, which can result in pipeline failures and eventually the Max retry reached error.
Resolution
Do not use Sparkflows macros directly within pipeline nodes when their values change frequently.
Instead:
- Define the required values as Sparkflows pipeline parameters.
- Pass the macro values to these pipeline parameters at runtime.
- Reference the pipeline parameters within the pipeline nodes.
This ensures that the generated DAG remains consistent across executions, preventing unnecessary DAG regeneration and avoiding the Max retry reached error.