EMR Cluster Job Failure: Unable to Locate ‘delta’ Data Source

General
1

Problem

While running a job on the EMR cluster, the following error is encountered:

Failed to find the delta data source

This indicates that the required Delta Lake JAR is either:

  • Missing from the EMR cluster environment.
  • Not passed correctly during job execution.
  • Incompatible with the Spark/Delta Lake version being used.

Cause

The issue occurs when the Delta Lake dependency is not available in the Spark runtime classpath or when there is a version mismatch between:

  • Apache Spark version.
  • Delta Lake JAR version.
  • Delta Lake API used by the application.

Solution

  1. Identify the compatible Delta Lake JAR version based on the Spark version running on the EMR cluster.
  2. Copy the required Delta JAR files to the EMR environment.
  3. Pass the Delta JAR files through the Livy connection configuration in Fire.
  4. Ensure multiple JAR files are provided as a comma-separated list in the Fire JAR configuration.

Example:

/path/to/delta-core_2.12-x.x.x.jar,/path/to/delta-storage-x.x.x.jar
  1. Restart the Spark session/job and verify that the Delta data source is successfully loaded.

Note

Refer to the official Delta Lake documentation to verify the supported Delta Lake version for your Spark version: