Job failed, when saving the model in S3

Following is the exception I received.

org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://mlflow-artifacts/prod/All_Models_Test/h2o_S3_test/6449cf60-a992-423a-bec1-aaba2643891a: com.amazonaws.SdkClientException: Unable to execute HTTP request: Certificate for <10.1.78.16> doesn't match any of the subject alternative names: [*.[ura.go.ug](http://ura.go.ug/), [ura.go.ug](http://ura.go.ug/)]: Unable to execute HTTP request: Certificate for <10.1.78.16> doesn't match any of the subject alternative names: [*.[ura.go.ug](http://ura.go.ug/), [ura.go.ug](http://ura.go.ug/)]
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:210)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3667)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3570)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$33(S3AFileSystem.java:4549)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2420)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2439)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4547)
at org.apache.spark.ml.util.FileSystemOverwrite.handleOverwrite(ReadWrite.scala:675)
at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:167)
at fire.nodes.h2o.NodeH2OMojoSave.execute(NodeH2OMojoSave.scala:87)
at fire.workflowengine.Node.executeTop(Node.java:236)
at fire.workflowengine.Node.executeNextNodes(Node.java:932)
at fire.workflowengine.Node.executeNextNodes(Node.java:815)
at fire.nodes.h2o.NodeH2OGbm.execute(NodeH2OGbm.scala:828)
at fire.workflowengine.Node.executeTop(Node.java:236)
at fire.workflowengine.Node.executeNextNodes(Node.java:932)
at fire.workflowengine.Node.executeNextNodes(Node.java:815)
at fire.workflowengine.Node.passDataFrameToNextNodesAndExecute(Node.java:1141)
at fire.nodes.etl.NodeColumnFilter.execute(NodeColumnFilter.java:80)
at fire.workflowengine.Node.executeTop(Node.java:236)
at fire.workflowengine.Node.executeNextNodes(Node.java:932)
at fire.workflowengine.Node.executeNextNodes(Node.java:815)
at fire.workflowengine.Node.passDataFrameToNextNodesAndExecute(Node.java:1141)
at fire.nodes.dataset.NodeDatasetParquet.execute(NodeDatasetParquet.java:86)
at fire.workflowengine.Node.executeTop(Node.java:236)
at fire.workflowengine.Workflow.executeNodes(Workflow.java:839)
at fire.workflowengine.Workflow.execute(Workflow.java:724)
at fire.execute.WorkflowExecuteUI.executeWorkflow(WorkflowExecuteUI.java:655)
at fire.execute.WorkflowExecuteUI.execute(WorkflowExecuteUI.java:236)
at fire.execute.WorkflowExecuteLivy.execute(WorkflowExecuteLivy.java:165)
at fire.execute.WorkflowExecuteLivy.main(WorkflowExecuteLivy.java:138)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Certificate for <10.1.78.16> doesn't match any of the subject alternative names: [*.[ura.go.ug](http://ura.go.ug/), [ura.go.ug](http://ura.go.ug/)]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1402)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2590)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2578)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2558)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3642)
... 37 more

The issue is caused due to SSL/TLS certificate hostname validation failure between the Hadoop S3A client (AWS SDK) and your S3-compatible object storage.
Since 10.1.78.16 is not listed in the certificate, the connection is rejected.

In order to fix the issue, you need to add the correct SAN entries for that certificate.