Spark Cluster Workflow Execution: No Browser Results

General
1

Problem

When executing workflows on a Spark cluster, the workflow completes or starts running, but no results are displayed in the Browser.

Cause

Sparkflows submits workflow jobs using spark-submit. The Spark driver is responsible for posting the execution results back to the Sparkflows (Fire) server.

If the Spark job fails, or if the driver cannot communicate with the Fire server, the Browser will not display the workflow results.

Resolution

1. Check Workflow Logs

Review the workflow execution logs to identify any Spark job failures.

Log Location:

/tmp/fire/workflowlogs

Look for errors related to:

  • Spark job failures
  • Driver exceptions
  • Network connectivity issues
  • Result posting failures

2. Verify Spark Job Submission

Ensure Spark jobs can be submitted successfully from the machine where Sparkflows is installed.

Run the SparkPi example to validate the Spark cluster.

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode client \
  spark-examples.jar 10

Note: spark-examples.jar is available in your Apache Spark installation directory.

If the SparkPi job fails, resolve the Spark cluster configuration issues before troubleshooting Sparkflows.

3. Verify Result Callback Configuration

If the Spark job completes successfully but no results appear in the Browser, verify that the Spark driver can post results back to the Sparkflows server.

Navigate to:

Administration → Configuration

Verify the following property:

Configuration Description
app.postMessageURL URL used by the Spark driver to send workflow execution results back to the Sparkflows (Fire) server.

Ensure:

  • The URL is correct.
  • The Fire server is reachable from the Spark driver node.
  • Firewalls or proxy settings are not blocking communication.

Validation Checklist

  • Verify workflow logs under /tmp/fire/workflowlogs.
  • Confirm Spark jobs run successfully using the SparkPi example.
  • Ensure app.postMessageURL is configured correctly.
  • Verify network connectivity between the Spark driver and the Sparkflows server.
  • Review Spark driver logs for callback or HTTP connection errors.

Expected Outcome

After correcting the Spark job execution issues or fixing the app.postMessageURL configuration, workflow execution results should be successfully posted back to the Sparkflows server and displayed in the Browser.