How Feature Engineering, Data Profiling and Data Cleansing can be performed in Sparkflows?

FAQs
1

Following nodes in Sparkflows can help to perform Data Profiling:

  1. Correlation - It displays relation between dependent and independent features. Relation between features is plotted in Heatmap Graph.
  2. Summary - It calculates and prints spreads of feature such as Count, Mean, Min, Max and so on.
  3. Using various ML Model nodes we can also get an insight into the importance of each feature.
  4. Flag Outlier - It flags outliers in the dataset.

Following nodes in Sparkflows can help to perform Data Cleansing:

  1. Imputing - There are various imputing nodes available to handle missing values. Using these nodes missing values can be replaced with either a Constant or Mean/Median/Mode value.
  2. Dedup - To resolve duplicate entity data.
  3. Drop Duplicate Rows - Handles duplicate rows.
  4. Null Value handling - There are various nodes to handles null values in the dataset.
  5. Find And Replace - There are various nodes to handle unwanted characters, replacing a string pattern with others and so on.

Following nodes in Sparkflows can help to perform Feature Engineering:

  1. String Indexer - It encodes String categorical data to numeric values.
  2. Min Max Scaler And Standard Scaler - They scale incoming data by reducing variance.
  3. Feature Extraction nodes
  4. Feature Transformation nodes
  5. Feature Selection nodes
  6. Splitting Dataset nodes