SparkML workflow with xgboost are failing

General
1

Problem

SparkML workflows using XGBoost may fail with the following error:

java.lang.Exception: Error: Failed to execute the workflow:
/tmp/libxgboost4j14342423119254343105.so: libgomp.so.1:
cannot open shared object file: No such file or directory

This indicates that the GNU OpenMP runtime library (libgomp.so.1) required by XGBoost is missing from the machine.

Solution

Ensure that the libgomp package is installed on the machine where the SparkML workflow is executed.

Ubuntu / Debian

Run:

sudo apt update
sudo apt install --reinstall libgomp1

CentOS / RHEL

Run:

sudo yum install libgomp -y

For newer RHEL-based systems using dnf, you can also run:

sudo dnf install libgomp -y

Verification

After installing the package, verify that libgomp.so.1 is available:

ldconfig -p | grep libgomp

Expected output should contain an entry similar to:

libgomp.so.1 (libc6,x86-64) => /lib64/libgomp.so.1

Once libgomp.so.1 is available, rerun the SparkML XGBoost workflow.