Problem
When submitting a job to a Databricks cluster from the Sparkflows UI, the job starts but no response is seen back in the Sparkflows UI.
Running the health check from the Databricks notebook also returns an empty response.
How can this be resolved?
Solution
This issue typically occurs due to network and load balancer configuration problems between Databricks and the Sparkflows endpoint used for postback events.
Sparkflows depends on the postback URL endpoint to receive execution updates from Databricks. If this endpoint is misconfigured at the infrastructure level, Sparkflows will not receive any response.
Root Causes
- The Sparkflows endpoint service used as the postback URL is in a different Availability Zone than the EC2 instance where Sparkflows is running.
- Cross-zone load balancing is not enabled on the endpoint service used for Sparkflows.
Best Practices
1. Ensure Same Availability Zone
The Sparkflows endpoint used for the postback URL must be in the same Availability Zone (AZ) as the EC2 instance hosting Sparkflows.
If the endpoint and EC2 instance are in different AZs:
- Health check calls may return empty responses
- Databricks postback communication may silently fail
Align the endpoint service and Sparkflows EC2 instance to the same AZ.
2. Enable Cross-Zone Load Balancing
If Sparkflows is accessed through an Endpoint Service / Load Balancer:
- Ensure cross-zone load balancing is enabled
- This allows traffic from Databricks to be routed correctly to the Sparkflows instance across availability zones
Without cross-zone load balancing:
- Health check calls can fail
- Databricks may report communication errors
- Sparkflows UI will not receive job status updates