You can achieve this by following these steps:
-
When splitting the data into training and test sets, make sure to save the test data along with the trained model.
-
In the subsequent steps, use the saved trained model to make predictions on the test data and save these predictions.
-
Create a PySpark workflow to read the predicted output and apply custom evaluation metrics to select the desired evaluation metrics.